JP2016115332A - Image processing device and control method and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of transforming three-dimensional form data for outputting a modeled object in a reduced scale or an enlarged scale suitable for the modeled object.SOLUTION: An image processing device includes: an acquisition means which acquires three-dimensional data describing space containing an object; a conversion means which respectively converts positional data of a predetermined direction of the three-dimensional data corresponding to a position where the object exists; and an output means which outputs the three-dimensional data in which the position of the predetermined direction is converted.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an image processing apparatus, a control method thereof, and a program, and more particularly to a three-dimensional data conversion technique.

  2. Description of the Related Art Conventionally, a method for measuring the distance to each pixel area of a captured image and acquiring three-dimensional data is known. For example, Patent Document 1 discloses a method for effectively acquiring three-dimensional data by combining distance measurement with a laser and a captured image.

JP 2004-037396 A

  By the way, in recent years, a technique called 3D printing or the like, which can output the three-dimensional shape of an object as a modeled object when inputting the three-dimensional data of the object, is becoming widespread. When the output of a three-dimensional shaped object becomes easy and spreads to homes, etc., with a certain type of three-dimensional shape data, it will meet the needs just by changing the scale or scale of the original data as it is. It is possible that there is no output object.

  The present invention has been made in view of the above-described problems of the prior art, and an image processing apparatus capable of transforming three-dimensional shape data for outputting a modeled object at a scale or a scale suitable for the modeled object, and An object is to provide a control method and a program thereof.

  In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is, an acquisition unit that acquires three-dimensional data describing a space including an object, a conversion unit that converts the position data of the three-dimensional data in a predetermined direction according to the position where the object exists, and a predetermined direction Output means for outputting the three-dimensional data in which the position of is converted.

  According to the present invention, three-dimensional shape data for outputting a modeled object can be deformed at a scale or a scale suitable for the modeled object.

The figure explaining the subject which should apply the conversion processing of three-dimensional data concerning the present invention 1 is a block diagram illustrating a functional configuration example of a digital camera 200 as an example of an image processing apparatus according to a first embodiment. 3 is a flowchart showing a series of operations for three-dimensional data conversion processing according to the first embodiment. The figure explaining the detection process of the range of the main object which concerns on Embodiment 1. The figure explaining compression of the depth direction of the three-dimensional data which concerns on Embodiment 1. FIG. Example of data format of three-dimensional data according to Embodiment 1 The figure explaining compression of the depth direction of the three-dimensional data by the polygon information which concerns on Embodiment 1. FIG. The figure explaining compression of the depth direction of the three-dimensional data as a modification which concerns on Embodiment 1. FIG. 9 is a flowchart showing a series of operations for three-dimensional data conversion processing according to the second embodiment. FIG. 10 is a diagram for explaining detection processing of a range of a main object and a sub main object in Embodiment 2. The figure explaining compression of the depth direction of the three-dimensional data which concerns on Embodiment 2. FIG. 9 is a flowchart showing a series of operations for three-dimensional data conversion processing according to the third embodiment. The figure explaining the detection process of the range of the main object which concerns on Embodiment 3. The figure explaining compression of the depth direction of the three-dimensional data which concerns on Embodiment 3. The figure explaining the correction process of the magnitude | size of the depth direction and perpendicular | vertical plane direction which concerns on Embodiment 3. FIG. FIG. 6 is a diagram for explaining a main object range detection method according to the third embodiment.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, an example in which the present invention is applied to a digital camera having a configuration capable of measuring a three-dimensional shape of a subject (also referred to as 3D scanning) will be described as an example of an image processing apparatus. However, the image processing apparatus referred to in the present invention is not limited to a configuration capable of measuring a three-dimensional shape of a subject, and can be applied to any electronic device that can acquire measured three-dimensional shape data. These electronic devices may include, for example, a mobile phone, a game machine, a tablet terminal, a personal computer, a clock-type or glasses-type information terminal.

(Configuration of digital camera 200)
FIG. 2 is a block diagram illustrating a functional configuration example of a digital camera 200 as an example of the image processing apparatus of the present embodiment. Note that one or more of the functional blocks shown in FIG. 2 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or by executing software by a programmable processor such as a CPU or MPU. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The imaging unit 201 includes a plurality of imaging optical systems and imaging elements, and outputs parallax images obtained by the imaging optical systems and imaging elements. The image sensor provided in the imaging unit 201 receives a subject optical image formed by one imaging optical system by a plurality of photoelectric conversion elements constituting a plurality of pixels arranged two-dimensionally, and each pixel The signal is photoelectrically converted to an analog signal. The image sensor may be an image sensor such as a complementary metal oxide semiconductor (CMOS) image sensor. The image sensor outputs an analog signal for each pixel in accordance with the timing signal supplied from the timing generator. The analog signal of each image output from each image sensor is A / D converted and output as digital image data (parallax image data) having parallax. The control unit 206 acquires parallax information using the parallax image data output from the imaging unit 201, and based on the parallax information, three-dimensional shape data of an object that is a subject (hereinafter also simply referred to as three-dimensional data). Is calculated. Note that one imaging element may be configured, for example, such that each pixel of all effective pixels is configured by a plurality of independent photodiodes and receives light beams from a plurality of different pupil regions of the imaging optical system. The image data corresponding to the light fluxes of different pupil regions is image data having parallax with each other, and the data in the depth direction can be acquired from a pair of image signals corresponding to the different pupil regions, similarly to the parallax image data. . In this way, parallax image data can be obtained using one imaging optical system.

  The screen display unit 202 displays an image output from the imaging unit 201 during acquisition of the three-dimensional data, and confirms information on already acquired shape data (existing scan information), a captured image, and the like.

  The screen operation unit 203 is disposed on the display surface of the screen display unit 202, for example, and notifies the control unit 206 of detection of an operation instruction when detecting a contact operation, for example, by the user. In response to the notification from the screen operation unit 203, the control unit 206 controls the screen display unit 202 in order to change a display such as a menu display.

  The object detection unit 204 performs object detection processing described later on the image data obtained by the imaging unit 201. In the object detection process, a calculation process is performed to detect a person or an object in the image and specify the position and size thereof, and the calculation result is output to the control unit 206.

  The button operation unit 205 detects an operation instruction from the user for the operation button or the like, and notifies the control unit 206 of the detection of the operation instruction or the content of the detection. The control unit 206 controls each unit of the digital camera 200 in response to a notification from the button operation unit 205.

  The control unit 206 is, for example, a CPU or MPU, and controls the entire digital camera 200 by developing and executing a program stored in the ROM 207 in the work area of the RAM 208. The control unit 206 calculates three-dimensional data based on the parallax information obtained from the parallax image data, and performs conversion processing of three-dimensional data described later on the calculated three-dimensional data.

  The ROM 207 is a non-volatile storage device composed of a semiconductor memory, a magnetic disk, or the like, and stores programs executed by the control unit 206 and setting values set in the digital camera 200.

  The RAM 208 is a volatile storage device that stores temporary data that may be erased after the digital camera 200 is turned off. The RAM 208 plays a role of storing temporary data being processed.

  The memory slot 209 is an interface for reading / writing recording data from / to the memory card 210. The memory card 210 is a recording medium composed of magnetism or a semiconductor, and recorded images, three-dimensional data, and setting values for the digital camera 200 are recorded via the memory slot 209.

  The acceleration sensor 211 and the gyro sensor 212 respectively detect the movement and rotation of the digital camera 200 itself. The control unit 206 uses information about the detected movement amount and rotation amount, and unidirectional object shape information that can be calculated from the output of the imaging unit 201 (that is, three-dimensional information obtained from the parallax image data) from multiple viewpoints. In combination with the acquired information, three-dimensional object data can be generated. In this embodiment, the parallax image data is used as a means for acquiring information in the depth direction (Z-axis direction). However, the present invention is not limited to this, and information on the depth direction, such as a TOF sensor using infrared rays, is used. The means for obtaining may be another method.

  The functional blocks of the imaging unit 201 to the gyro sensor 212 are connected to each other by a bus and can transmit and receive data.

(Necessity of 3D data conversion processing according to this embodiment)
With reference to FIG. 1, the example of the subject in the case of outputting the three-dimensional data obtained by carrying out 3D scanning of the space comprised by a subject as a molded article is demonstrated. FIG. 1A shows the positional relationship between the digital camera 200 and the real objects 101 to 104 that are the subjects. The objects 101 to 104 exist at respective positions in the depth direction (Z-axis direction) from the digital camera 200, and each has an object-specific depth (thickness). The object 101 and the object 102 are located at a predetermined distance from the Z axis in the plus X direction and the minus X direction, respectively, and are located at the same distance from the digital camera 200 in the Z axis direction. An object 103 and an object 104 are located at a predetermined distance away from the object 101 and the object 102 (in the plus Z-axis direction). It is assumed that 3D data of such objects 101 to 104 is 3D scanned by the digital camera 200. Here, as in this embodiment, two-dimensional data in the X-axis direction and the Y-axis direction are acquired from image signals acquired by an image sensor having photoelectric conversion elements arranged two-dimensionally, and depth information is further acquired. The three-dimensional data generated by the configuration to be described will be described. For example, when imaging is performed with an imaging element included in the imaging unit 201, the size of the subject in the captured image is captured smaller than the actual size according to the distance from the digital camera 200. Therefore, even if the three-dimensional data is generated by simply adding the data in the Z-axis direction of the depth information to the two-dimensional data (image data) obtained from the imaging unit 201, the relationship of the size of each subject in the real space is reproduced. Not necessarily. In order to model the three-dimensional data in the real space, it is necessary to change the size of each subject in the XY directions based on the depth information. However, in the present embodiment, it is assumed that a user who has data that can be viewed as two-dimensional image data seeks to make the image three-dimensional and outputs a shaped object. For this reason, the three-dimensional data provided to the 3D output device does not necessarily have to be three-dimensional data reproduced to the relationship of the size of each subject. In some cases, the former three-dimensional data equal to the image data is preferable. Therefore, in this embodiment, it is assumed that both data in the XY directions can be considered as three-dimensional data describing a space including an object.

  Regarding the three-dimensional data obtained by the above-described 3D scanning, information on the depth direction will be further considered. The depth information in the obtained three-dimensional data is changed (normalized) to a size that can be output by the 3D output device, for example, as shown in FIG. The objects 109 to 112 in the three-dimensional data correspond to the objects 101 to 104, respectively. In each object shown in FIG. 1B, the Z-axis direction is uniformly reduced in accordance with the range of the 3D output device, and the object 101 to 104 is reduced while maintaining the positional relationship in the depth direction.

  However, when the photographed three-dimensional data is output as a memorial object or the like, for example, as shown in FIG. 1C, the scale in the depth direction is appropriately reduced according to the characteristics of the object, or as shown in FIG. Thus, there is a need for outputting as a relief-like 3D output object. Further, in the output of the three-dimensional shape data of the type as shown in FIG. 1B, the space between the object 111 and the object 112 is wasted, or the object 111 or the like that existed in the image data is used. In some cases, an important subject is not emphasized and cannot be viewed impressively, and the configuration does not match the user's intention. Therefore, it is not always necessary to accurately reproduce the positional relationship and depth of each object, but it is desirable to perform a three-dimensional data conversion process that changes the depth between objects as necessary.

  Therefore, in this embodiment, for example, the scale of data at a position in a predetermined direction such as the z-axis direction is changed, and the scale (compression ratio) can be changed according to the position (range) in the z-axis direction. That is, in this embodiment, the data of the position in the predetermined direction of the three-dimensional data is converted according to the position where the object exists. For example, as shown in FIGS. 1C and 1D, the scale of the depth portion between the object 111 and the object 112 is made larger than the scale of the range of the object 111 or the object 112. That is, in the present embodiment, for example, the scale in the depth direction is changed according to the position in the depth direction by the conversion process of three-dimensional data described below. The scale in the z-axis direction may be performed independently of the scale in the x-axis direction and the y-axis direction orthogonal to the z-axis direction, or may be associated with something.

(A series of operations related to conversion processing of three-dimensional data)
Next, with reference to FIG. 3, a series of operations related to the conversion process of the three-dimensional data by the digital camera 200 will be described. This process is started when, for example, a shooting instruction from the user to the button operation unit 205 of the digital camera 200 is detected. Note that each step related to this processing is executed in accordance with an instruction from the control unit 206, and is realized by the control unit 206 developing and executing a program stored in the ROM 207 in the work area of the RAM 208.

  In step S301, the control unit 206 performs a 3D scan for acquiring the parallax image data output from the imaging unit 201, the output of the acceleration sensor 211, and the output of the gyro sensor 212, and a three-dimensional data shaping process based on the acquired data. Do. In the present embodiment, an example will be described in which the objects 101 to 104 to be subjects shown in FIG. 1A are targeted for 3D scanning. In the present embodiment, scanning from multiple directions is performed with reference to an image at a certain position and direction (for example, the digital camera 200 in FIG. 1A), that is, photographing is performed a plurality of times from a plurality of directions. At the time of shooting, a change in shooting position and orientation of the digital camera 200 is acquired in association with each shooting, and the three-dimensional shape of the subject is obtained from the obtained data. As a method for obtaining a three-dimensional shape of an object as a subject based on a plurality of parallax image data and photographing position and orientation fluctuations, a known method can be used. For example, if the information of the acceleration sensor 211 and the gyro sensor 212 is used, the positional relationship between a plurality of positions where photographing has been performed can be grasped. Therefore, the point cloud data as all 3D data can be converted into positions in one 3D space. Can be dropped. By this combining operation, 3D point cloud data of the surface shape including the back and front of the subject can be acquired. After scanning the three-dimensional shape of the object shown in FIG. 1A, the control unit 206 performs a three-dimensional data forming process, that is, a process for forming general three-dimensional data such as an STL file format.

  The STL file format uses a method of expressing a three-dimensional shape as a set of triangle faces, and it is necessary to convert the three-dimensional point cloud data into triangle set data. For this conversion, several methods are generally known. In this embodiment, a three-dimensional Delaunay division method, which is a widely known algorithm, is used. The algorithm of the Delaunay division method is stored in advance in the ROM 207, and the point cloud data of the object obtained by the above procedure is converted into a triangular plane set, which is converted into a triangular shape in a three-dimensional space. Is converted into the STL file format which is the aggregate data.

  In this embodiment, it is assumed that the scale processing is changed to a size that can be performed by the 3D output device by the molding process in this step, and three-dimensional data in which each of the x, y, and z axes is reduced at the same scale is generated. explain. When the data shaping process ends, the control unit 206 stores the shaped data in the memory card 210 via the memory slot 209. In this embodiment, a 3D scan using a plurality of imaging optical systems and parallax image data thereof will be described. However, as long as three-dimensional data is obtained, a mode using laser irradiation, a mode using an image plane phase difference sensor, and the like. This form is also applicable. When the image plane phase difference detection mechanism is mounted on the imaging unit 201, distance data can be obtained in units of pixels. Therefore, the image data obtained by performing the imaging process from a specific position includes color data for each pixel in the image and, in addition, the distance from the digital camera 200 to the surface portion of the subject corresponding to each pixel for each pixel. Information can be acquired. The point cloud data in which the distance in the depth direction is arranged for each pixel is raw 3D data that can be acquired.

  In S302, the control unit 206 determines a coordinate axis to define the depth direction. In the present embodiment, for example, the z direction is defined as the depth direction. The y direction represents a vertical direction with respect to the ground, and is defined by specifying the direction of gravitational acceleration by the acceleration sensor 211 during 3D scanning. Since the digital camera itself faces in various directions during 3D scanning, it is not determined which one is in the z-axis direction immediately after scanning. For this reason, the control unit 206 displays the outline of the three-dimensional data of the scan result on the screen display unit 202, and determines the z-axis direction that is the depth direction by the user via the user interface of the screen operation unit 203. When the z-axis direction is determined, the y-axis direction is determined according to the definition of the y-axis described above, and the x-axis direction is further determined. The control unit 206 converts the three-dimensional data generated in S301 into the three-dimensional data of the newly determined coordinate system and stores it in the memory card 210. In the following description, it is assumed that the z-axis shown in FIG.

  In addition, an image corresponding to a predetermined cross section of the three-dimensional data is determined from the two-dimensional captured image stored during the 3D scan. In other words, one image taken in the determined z-axis direction and having the above-described objects 101 to 104 within the angle of view is determined and stored in the memory card 210 as additional information of three-dimensional data. .

  In the present embodiment, since it is assumed that scanning (that is, photographing) is performed from multiple directions, the user sets the z-axis direction. However, in the case of a scanning method in which back information is not acquired only by scanning from one direction, the scanning direction may be set as the z axis as it is. In this case, the user does not need to specify the z axis.

  In step S <b> 303, the control unit 206 determines a main object to be noted among objects that are subjects. FIG. 4A shows an example of the two-dimensional image determined in S302 and an object detection result for the image. The object detection unit 204 detects an object in the image in the two-dimensional image in accordance with an instruction from the control unit 206, and then the control unit 206 displays the detection result by the object detection unit 204 on the screen display unit 202. Note that since a known object detection technique can be used for detecting an object in a two-dimensional image, a detailed description of the object detection technique is omitted. In the display example in FIG. 4A, object detection frames 405 to 408 for clearly indicating that each of the four objects in the detected image, that is, the objects 401 to 404 are detected, are displayed. Yes. The user specifies a main object by selecting one of the object detection frames 405 to 408 by an instruction operation on the screen operation unit 203. When a predetermined object is specified by a user operation, the control unit 206 determines the specified object as a main object. Note that the main object determination method is not limited to the above-described manual determination method by the user, and the control unit 206 may automatically determine the main object. In this case, it may be determined by information such as the position of the object within the angle of view and the characteristics of the object such as whether the object is a person.

  In step S304, the control unit 206 detects the thickness of the determined main object in the depth direction. The position and orientation at which the image shown in FIG. 4A is taken can be obtained from the positional relationship recorded during the 3D scan, and is the position and optical axis orientation of the digital camera 200 shown in FIG. 4B. Shall. The control unit 206 can specify how the object detection frame 407 in FIG. 4A is in the acquired three-dimensional data. More specifically, FIG. 4B shows what position the object detection frame 407 is in the three-dimensional data. The control unit 206 scans the object detection frame 407 in the z-axis direction, and detects three-dimensional data of the object region included in the rectangular parallelepiped 410 formed by the scanning.

  More specifically, for example, in the case of the STL file format or the like, the three-dimensional data is expressed as a collection of triangular information on a three-dimensional minimum unit called a polygon. First, the control unit 206 detects all the polygons included in the above-mentioned rectangular parallelepiped. In addition, in consideration of the connection status between the detected polygons, whether the polygon group forms a closed curved surface in the above-mentioned rectangular parallelepiped 410, or the mapping of the range formed by the closed curved surface to the xy plane is performed. It is determined whether the object detection frame 407 occupies a certain amount or more. If both determinations are satisfied, the control unit 206 determines the three-dimensional data included in the rectangular parallelepiped 410 as the object region of the main object. On the other hand, when a plurality of objects composed of polygonal closed curved surfaces are detected when the above-described conditions are satisfied, for example, three-dimensional data relating to one object is determined by adding a condition such as “in front of”. To do. In the case of this embodiment, the polygon information which comprises the object 103 is extractable by performing such a process. It is calculated by estimating the thickness of the object 103 using the difference between the smallest z-axis component and the largest z-axis component of the detected polygon information constituting the object 103. The calculated thickness is the object thickness 409 in FIG.

  In S305, the control unit 206 generates a compression profile function H (z) for the z-axis direction based on the position and thickness in the z-axis direction of the object 103 calculated in S304. Specifically, the compression profile function H (z) is a function of z indicating how much the component at which position on the z-axis is to be compressed, and represents a conversion characteristic in the z direction. The compression profile function H (z) is generated using the position and thickness in the z-axis direction of the object 111 calculated in S304. With reference to FIG. 5, compression in the z-axis direction including generation processing of the compression profile function H (z) will be described. In the following description, the z direction will be described for the sake of simplicity, but the x and y directions are converted at a compression rate of 1, for example.

  FIG. 5A schematically shows the positional relationship in the three-dimensional data viewed from the direction perpendicular to the z axis (the positive side of the x axis). FIG. 5B shows the characteristics of the compression profile function H (z). The horizontal axis indicates the depth (z), and the vertical axis indicates the compression rate. For example, when the compression ratio is 1, the component in the z-axis direction is maintained as it is. For example, when the compression ratio is 2, the distance in the z-axis direction is further compressed to ½. For the object 111, the position and thickness do not change because the compression ratio of the portion corresponding to the position where the object 111 exists and the thickness range is 1. On the other hand, the positions other than the object 111 show a characteristic in which the compression rate increases linearly with distance from the position. For this reason, the positional relationship in the z-axis direction between the object 111, the object 109, and the object 112 is respectively compressed, and between the objects 501 and 502 and between the objects 502 and 503, as shown in FIG. The distance is shortened. Further, since the thicknesses of the object 109 and the object 112 are also compressed, respectively, the thicknesses are compressed to the thicknesses of the objects 501 and 503 shown in FIG. As described above, the control unit 206 keeps the compression rate low with respect to the main object based on the position and thickness of the main object in the z-axis direction, and the compression rate increases linearly according to the distance from the main object (that is, monotonously increases). To generate a compression profile function.

  In S306, the control unit 206 generates a conversion function G (z) for actually converting z-axis data of the three-dimensional data from the compression profile function. When expressed by a continuous equation, the conversion function G (z) can be generated from the compression profile function H (z) by the equation represented by Equation 1. Formula 1 is a discrete formula suitable for CPU calculation or the like and is expressed by Formula 2.

  The compression profile function H (z), conversion function G (z), and z ′ after conversion in the z-axis direction will be described with reference to FIG. For simplicity of explanation, it is assumed that the horizontal axis z is normalized to 0-100. The position of z = 0 corresponds to the forefront position of the data range in the z-axis direction, and the position of z = 100 corresponds to the rearmost position of the data range in the z-axis direction. FIG. 5D shows a compression profile function H (z), and compression ratios other than the main object range (z = 30 to 40) can be arbitrarily set. The compression ratios other than the range of the main object may be calculated from the relationship between the position and thickness of the main object, for example, with the compression ratios at the forefront position and the rearmost position in the z-axis direction as predetermined compression ratios. Good. The conversion function G (z) shown in FIG. 5 (e) is a function calculated by applying Equations 1 and 2 to the compression profile function H (z) shown in FIG. 5 (d). If the coordinate before compression conversion is z and the coordinate after conversion corresponding to z is z ′, the relationship between z and z ′ is expressed by Equation 3. FIG. 5F is a graph showing the relationship between z and z ′.

  In S307, the control unit 206 performs a process of converting the data in the z-axis direction of the three-dimensional data obtained in S302 using the conversion function G (z) in the z-axis direction determined in S306. FIG. 6 shows the syntax of an STL (STereoLithography) file format, which is an example of a three-dimensional data file format. As shown in FIG. 6, the three-dimensional data is expressed in a format in which polygon information that is triangle information in a three-dimensional space is listed. More specifically, the “facet normal” line to the “endfacet” line constitute specific data of triangle data, and the specific data is described in the “facet normal” line and the “vertex” line. The “facet normal” line represents the normal vector of the triangle, the three “vertex” lines represent the coordinates of each vertex of the triangle data, and each vertex has numerical values of x, y, and z components. Accordingly, the control unit 206 converts the position in the z-axis direction by applying the conversion function G (z) to the information on each vertex of the triangle.

  FIG. 7 shows how polygon triangle information is converted in the z-axis direction using the z-axis conversion function G (z). The polygon information indicated by S in FIG. 7A is composed of three vertices of A (x1, y1, z1), B (x2, y2, z2) and C (x3, y3, z3). The control unit 206 does not change the x component and the y component with respect to all polygon information included in the three-dimensional data, calculates z ′ represented by Equation 3 for only the z component, and changes z by z ′. Create the data. FIG. 7B shows polygon information in which S shown in FIG. 7A is S ′ after only the z component is compressed by Equation 3. Each vertex information of A to C constituting S is converted into A ′ (x1, y1, z1 ′), B ′ (x2, y2, z2 ′), C ′ (x3, y3, z3 ′), respectively. Yes.

  When the conversion processing for all the polygon information is completed, the control unit 206 stores the converted polygon data listed in the STL format in the memory card 210 via the memory slot 209. When the control unit 206 stores the converted three-dimensional data, the control unit 206 ends a series of operations related to the three-dimensional data conversion process.

  In the present embodiment, the three-dimensional data obtained by the 3D scan in S301 is preliminarily reduced at the same scale in the x, y, and z directions, and further converted in the z-axis direction. did. However, in S301, a scale obtained based on the compression profile function without performing a reduction at the same scale in advance and a scale different from the x-axis and y-axis directions may be applied to the z-axis direction. Good. In this case, if the compression profile function H (z) is a function representing a scale for an actual object, the processing of this embodiment can be realized.

  In this embodiment, in the compression profile function H (z), in the region other than the main object, the depth direction of the three-dimensional data is converted by changing the compression rate linearly. However, the compression profile function H (z) can have various characteristics according to the user's intention and the characteristics of the object. For example, modified examples of H (z) are shown in FIGS. 8A is the same as FIG. 5A described above, but FIG. 5B determines how much the distance in the z-axis direction of the three-dimensional data shown in FIG. 8A is compressed. The characteristic of the compression profile function H (z) is shown. In H (z) shown in FIG. 8B, the compression rate is increased in a staircase pattern from the end position on the z-axis of the main object. By doing so, as shown in FIG. 8C, while maintaining the positional relationship between the converted object 901 and the object 902 (main object), the positional relationship and The depth can be greatly reduced. Another modification is shown in FIGS. 8D to 8F. FIG. 8 (d) is similar to FIG. 8 (a), but FIG. 8 (e) further illustrates the start of the main object on the z-axis in the compression profile function H (z) shown in FIG. 5 (b). The compression rate is raised and lowered in a staircase before and after the position and end position. In this way, the positional relationship and the depth of the object 1002 (main object) before and after the object 1002 (main object) are reduced at the same magnification while maintaining the thickness of the converted main object. In this case, the main object can be emphasized while matching the distance between the objects 1001 and 1002 and the distance between the objects 1002 and 1003 with the positional relationship of the real objects.

  As described above, in the present embodiment, the scale for the data in a predetermined direction such as the z-axis direction is changed, and the scale (compression ratio) can be changed according to the position (range) in the z-axis direction. . That is, the compression profile function H (z) is defined for the depth direction, and the data at the position in the depth direction of the three-dimensional data is converted according to the position where the object exists. If it does in this way, it will become possible to deform | transform the three-dimensional data for outputting a molded article with the reduced scale or double scale suitable for a molded article.

  Also, when changing the scale according to the position in the depth direction, depending on the type of object, that is, while reducing the compression rate in the depth direction with respect to the area of the main object, while increasing the compression ratio outside the area of the main object, Modified the depth of 3D data. That is, the overall compression rate in the depth direction and the compression rate in the depth direction of a specific subject are set. In this way, as shown in FIG. 5C, the main object can be emphasized or given presence to other objects, and a suitable shaped object can be used as a memorial object or interior object. Can be generated.

(Embodiment 2)
Next, Embodiment 2 will be described. In Embodiment 1, three-dimensional data conversion processing is performed using a compression profile function that lowers the compression rate of the selected main object and increases the compression rate of other regions. In this embodiment, three-dimensional data conversion processing is performed using a compression profile function with a reduced compression rate for a plurality of main objects, that is, sub-main objects other than the main object. For this reason, the process for determining the sub main object and the compression profile function to be determined are different from those in the first embodiment, and other configurations are the same as those in the first embodiment. For this reason, the same configurations and steps will be denoted by the same reference numerals, and redundant description will be omitted, and differences will be mainly described.

  The flow of processing in this embodiment will be described with reference to the flowchart shown in FIG.

  First, the control unit 206 performs the processing of S301 to S303 shown in FIG.

  In step S <b> 901, the control unit 206 determines a sub main object having higher priority than the main object selected in step S <b> 303. As shown in FIG. 4A, object detection frames 405 to 405 for clearly indicating that the four objects in the image detected by the process in S303, that is, the objects 401 to 404 are detected. 408 is displayed. The user selects one of the object detection frames other than the main object selected in S303 by operating the screen operation unit 203. Upon receiving a user operation notification from the screen operation unit 203, the control unit 206 determines the corresponding object as a sub main object. Here, it is assumed that the control unit 206 determines the object 112 as the sub main object in response to the user selecting the object detection frame 408. The sub main object may be determined not only by manual selection by the user but also by the control unit 206 automatically. In this case, the sub main object may be determined by determining where it is located within the angle of view and whether the object is a person. If necessary, a plurality of sub main objects may be designated.

  In step S902, the control unit 206 calculates the thickness in the z-axis direction for each of the main object and the sub main object. The control unit 206 calculates the thicknesses of the main object and the sub main object, respectively, in the same manner as described in S305. For example, as illustrated in FIG. 10, the control unit 206 scans the object detection frame 408 of the sub main object in the z-axis direction, and detects three-dimensional data included in the rectangular parallelepiped 1010 that has been scanned. At this time, for example, when another rectangular parallelepiped is detected, such as the rectangular parallelepiped 410, a rectangular parallelepiped having the highest ratio between the area of the xy plane of the rectangular parallelepiped and the object detection frame 408 may be selected.

  In step S903, the control unit 206 generates a compression profile function H (z) based on the positions and thicknesses of the main object and the sub main object calculated in step S902 in the z-axis direction. The compression profile function H (z) according to the present embodiment will be described with reference to FIG.

  FIG. 11A schematically shows the positional relationship of an object viewed from a direction perpendicular to the z-axis (plus x-axis direction). FIG. 11B shows a compression profile function that determines how much z-axis direction information of the object shape data is compressed as in FIG. In FIG. 11 (b), the compression ratio is set to 1 for the position and thickness range where the main object exists, as in FIG. 5 (b). On the other hand, the compression rate is uniformly set to 2 for the distance corresponding to the position and thickness range of the object 112 as the sub main object. Except for the positions and thickness regions of the object 111 and the object 112, the compression rate is set to increase linearly with distance from the position of the object 111.

  As described above, in the present embodiment, the control unit 206 sets the compression rate for the main object to be low and the compression rate to increase linearly according to the distance from the main object. A compression profile function H (z) is generated so that the compression rate becomes low. By doing so, as shown in FIG. 11C, in the reduced z ′, the thickness of the object 1102 (main main object) is maintained in addition to the thickness of the object 1102 (main main object). Maintained and reduced. That is, although the positional relationship between the objects 1101 to 1103 is reduced, the main object and the sub main object are relatively large, and the presence of these objects is emphasized. In addition, the compression ratio for the sub main object is set differently from the main object, so the sub main object is reduced at a slightly higher scale than the main object, and the main object is emphasized more than the sub main object. It has been adjusted so that.

  Next, the control unit 206 performs the processes of S306 and S307 illustrated in FIG. 3 and ends a series of processes related to this process. In addition, FIG.11 (e) has shown the conversion function G (z) which produced | generated the compression profile function H (z) shown in FIG.11 (d) produced | generated by the above-mentioned method by Numerical formula 1 and Numerical formula 2. FIG. . FIG. 11F shows the relationship between z converted by the conversion function G (z) and z ′ after conversion.

  In this embodiment, the compression profile function H (z) is linearly increased in the range other than the main object. However, the compression profile function H (z) is stepped from the end positions on the z axis of the main object and the sub main object. Alternatively, the compression rate may be increased. Alternatively, the compression ratio may be automatically generated so as to increase and decrease the compression rate in a stepwise manner before and after the start position and the end position on the z axis of the main object and the sub main object.

  In the compression profile function H (z), the compression rate in the range of the main object has been described as a predetermined value. However, the scale (or compression rate) in the depth direction may be set or changed according to the position by input to the user interface such as the screen operation unit 203. In this way, it is possible to generate a modeled object that emphasizes the main object at a scale desired by the user.

As described above, in this embodiment, three-dimensional data conversion processing is performed using the compression profile function in which the compression ratio of the main object and the sub main object is lowered as shown in FIG. . In this way, not only the main object but also the sub main object can be emphasized or given a presence, and it is possible to generate an appropriate shaped object as a memorial object, interior object, or the like.
(Embodiment 3)
Next, Embodiment 3 will be described. In the above-described embodiment, conversion processing of three-dimensional data is performed using a compression profile function that lowers the compression rate of a plurality of selected main objects (main object and sub-main object) and increases the compression rate of other regions. It was. In the present embodiment, in the conversion process of three-dimensional data, a process of correcting the size in the xy plane direction of a part of a plurality of main objects (for example, sub main objects) is further performed. For this reason, the processing and configuration other than the size correction processing are the same as those in the second embodiment. For this reason, the same configurations and steps will be denoted by the same reference numerals, and redundant description will be omitted, and differences will be mainly described.

  First, the relationship between the size and position of the main object and the sub main object assumed in the present embodiment will be described with reference to a part of FIG. 13A (details of FIG. 13A will be separately described later). In the present embodiment, it is assumed that the main object 1301 and the sub main object 1302 have an actual size of the main object 1301 that is, for example, half the actual size of the sub main object 1302. This is the case, for example, when the main object 1301 is a child and the sub main object 1302 is an adult. Further, it is assumed that the distance (L1) from the digital camera 200 to the main object 1301 is half of the distance (L2) from the digital camera 200 to the sub main object 1302. In this case, the two-dimensional image in the xy direction captured from the position of the digital camera 200 is, for example, as shown in FIG. That is, the actual size of the main object 1301 is half that of the sub main object 1302, but the same size on the two-dimensional image. If a final three-dimensional shaped object is output at this ratio, the size relationship may be unnatural because the object is not necessarily viewed from the direction and position shown in FIG. There is sex. Therefore, in the present embodiment, the size of the sub main object in the xy plane direction is normalized (ie, corrected) based on the size of the main object in the xy plane direction. Hereinafter, in the present embodiment, a correction process for generating a model in which the sizes of adults and children are more appropriately reflected will be described as an example.

(A series of operations related to conversion processing of three-dimensional data)
Next, a series of operations of the three-dimensional data conversion process according to the present embodiment will be described with reference to the flowchart shown in FIG.

  First, the control unit 206 performs the processing of S301 to S303 shown in FIG. 3 or FIG.

  In step S1201, the control unit 206 identifies a sub main object having a higher priority but having a lower priority than the main subject selected in step S303. The processing according to this step is the same as the processing of S901 described above, but will be described again with reference to FIG. In the two-dimensional image selected in S302, two objects (1301 and 1302) detected in S303 and object detection frames 1303 and 1304 that clearly indicate that each object has been detected are displayed. At this time, when the user selects an object detection frame other than the main object selected in S303 by operating the screen display unit 202, the control unit 206 determines an object corresponding to the selected object detection frame as a sub main object. To do. For example, when the user selects the object detection frame 1304, the sub main object 1302 is determined as the sub main object.

  In step S1202, the control unit 206 calculates the thickness in the z-axis direction for each of the main object and the sub main object. The control unit 206 calculates the thickness in the z-axis direction in the same manner as in the method shown in S305. For example, as shown in FIG. 13B, the control unit 206 scans the object detection frame 1303 for the main object 1301 (object 1307) and the object detection frame 1304 for the sub main object 1302 (object 1308) in the z-axis direction. To do. Then, the three-dimensional data included in the rectangular parallelepipeds 1309 and 1310 obtained by scanning is detected to detect the object thicknesses 1305 and 1306.

  In step S1203, the control unit 206 generates a compression profile function H (z) based on the position and thickness in the z-axis direction with respect to the object 1307 and the object 1308 calculated in step S1202. The compression profile function H (z) according to the present embodiment will be described with reference to FIGS. 14 (a), 14 (b), and 14 (d) to 14 (f) as appropriate.

  FIG. 14A schematically shows the positional relationship of objects viewed from a direction perpendicular to the z axis (plus x axis direction). FIG. 14B shows a compression profile function that determines how much the z-axis direction information of the object shape data is compressed, similar to the compression profile function in the above-described embodiment. In FIG. 14B, the compression ratio is set to 1 for the position and thickness range where the object 1307 as the main object exists, as in FIG. On the other hand, for the distance corresponding to the position where the object 1308 as the sub main object exists and the thickness range, the compression rate is uniformly set to 2. Except for the positions and thickness regions of the objects 1307 and 1308, the compression rate is set to increase linearly as the distance from the position of the object 1307 increases.

  As described above, in the present embodiment, the control unit 206 is set so that the compression ratio for the main object is low and the compression ratio is linearly increased according to the distance from the main object, while the sub main object is exceptionally set. A compression profile function H (z) is generated so that the compression rate becomes low. In this way, in the reduced z ′, the thickness of the object 1308 (main main object) is maintained and the thickness of the object 1308 (sub main object) is maintained and reduced.

  Next, the control unit 206 performs the processes of S306 and S307 shown in FIG. FIG. 14 (e) shows a conversion function G (z) generated by the expression 1 and the expression 2 from the compression profile function H (z) shown in FIG. 14 (d) generated by the above method. . FIG. 11F shows the relationship between z converted by the conversion function G (z) and z ′ after conversion.

  Further, in S1204 and S1205, the control unit 206 corrects the size of the sub main object in the xy plane direction with respect to the main object.

  First, in step S1204, the control unit 206 acquires a setting by the user who has selected whether or not to correct the size of the object 1308 (sub main object) in the xy plane direction. For example, this setting is used to select whether or not to correct the size of the object 1308 (sub-main object) in the xy plane direction in accordance with the size of the object 1307 (main object). If the acquired setting is a setting for performing correction, the control unit 206 advances the process to S1205. If the acquired setting is a setting for performing no correction, the control unit 206 ends the series of operations related to this processing without performing the correction.

  In step S1205, the control unit 206 corrects the size of the object 1308 (sub main object). More specifically, the control unit 206 calculates a parameter for correcting the size of the object using the ratio of the distance to each object. For example, FIG. 15 shows a schematic diagram of the yz plane when FIG. 13B is viewed from the x-axis direction. For example, it is assumed that the distance from the digital camera 200 to the main object 1301 is L1, the distance from the digital camera 200 to the sub main object 1302 is L2, and there is a relationship of L2 = L1 × 2. Further, it is assumed that the length of the main object 1301 in the y direction is H1, the length of the sub main object 1302 in the y direction is H2, and there is a relationship of H2 = H1 × 2. Furthermore, the angle of view of the digital camera 200 is θ. At this time, assuming that the vertical width corresponding to the range of the angle of view at the distance of L1 is the vertical angle of view angle 1, and similarly the vertical width corresponding to the range of the angle of view at the distance of L2 is vertical angle of view 2, the similar relationship Further, the angle of view length 2 is twice as large as the angle of view length 1. Therefore, the size of each object in the two-dimensional image when imaged from a certain imaging position is inversely proportional to the distance to the object. That is, the control unit 206 corrects (that is, changes magnification) the x-direction and y-direction sizes of the 3D data information of the object 1308 (sub-main object) using the ratio of L2 / L1, and sub-major the object 1402 Convert to object data. For example, FIG. 14C shows an object 1401 and an object 1402 that are the results of correcting the size of the object 1308.

  In this way, the size of the converted object 1401 and the converted object 1402 in the xy plane direction can be set to a more realistic size relationship. In addition, as in the second embodiment, the sub main object data of the object 1402 also becomes a model that is emphasized with a thickness as compared with the object (for example, 503) after the conversion by the method of the first embodiment.

  And the control part 206 will complete | finish a series of operation | movement which concerns on this process, if the sub main body data of the object 1402 are produced | generated.

  In S1204 described above, an example has been described in which whether or not to correct the size of the sub main object is selected and set. The reason why it is possible to select whether or not to perform correction in this way is that it may be more appropriate to turn correction off. For example, in the example shown in FIGS. 16A and 16B, the main object 2001 and the sub main object 2002 are photographed as subjects in the two-dimensional image, and the corresponding three-dimensional object 2003 (main object) and An object 2004 (sub main object) is shown.

  In this example, the main object 2001 is a person, and the sub main object 2002 is a huge landmark such as a tower. At this time, it is assumed that the distance L4 to the object 2004 is much longer than the distance L3 from the digital camera 200 to the object 2003. Also, the image shown in FIG. 16A has a composition in which, for example, a commemorative photo, the main object 2001 as a person and the sub main object 2002 as a tower are to be copied. In such a case, if the correction processing described in S1205 is performed, the 3D data of the sub main object 2002 is enlarged by an actual ratio and becomes excessively large, and the relationship of the intended subject when shooting FIG. May collapse. Therefore, it is desirable to be able to perform control so that the size of the sub main object is not corrected (that is, turned off) when photographing as shown in FIG. 16 is performed.

  That is, according to the present embodiment, it is possible to control the presence / absence of a correction process for the size of the main object 1301 and the sub main object 1302 in the xy plane direction according to the content of shooting. Even if this correction processing is not performed, as shown in FIG. 11 (c), the main object and the sub main object are emphasized, and the parts other than the object are compressed, souvenirs, interior objects, etc. Can be converted into data suitable for outputting a modeled object.

  In the present embodiment, the example in which the sub main object is determined by the user operation in S1201 has been described. However, for example, the control unit 206 may automatically determine one or more sub main objects based on the determination result such as the position within the angle of view of the sub main object and whether the object is a person.

  In the present embodiment, an example in which the selection of whether or not to perform correction processing is selected by a user operation in S1204 has been described. However, for example, the recognition of an object type, an object size, or the like is performed regardless of a user operation. Based on this, the presence / absence of correction processing may be automatically selected. For example, the control unit 206 may turn off the correction process when the difference in size between the main object and the sub main object is larger than a predetermined threshold. Further, for example, the control unit 206 may turn on the correction process when it is determined that both the main object and the sub main object are persons.

  As described above, in the present embodiment, the size of the sub main object in the xy plane direction is corrected based on the ratio of the distance between the main object and the sub main object in the depth direction. By doing in this way, about the three-dimensional shape data for outputting a modeling thing, the magnitude | size of the direction (xy plane direction) perpendicular | vertical to a depth direction is converted into a more realistic magnitude relationship, and is output. be able to. In addition, it is possible to set whether or not to perform this correction, and in particular, when the ratio of the distance in the depth direction between the main object and the sub main object is not in an appropriate relationship, the correction is not performed. By doing in this way, it becomes possible to deform | transform the three-dimensional shape data for outputting a molded article with the reduced scale or double scale suitable for a molded article.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 201 ... Imaging part, 202 ... Screen display part, 203 ... Screen operation part, 204 ... Object detection part, 205 ... Button operation part, 206 ... Control part

Claims (17)

Acquisition means for acquiring three-dimensional data describing a space including an object;
Conversion means for converting the position data of the three-dimensional data in a predetermined direction according to the position where the object exists,
And an output means for outputting three-dimensional data in which the position in the predetermined direction is converted.   The image processing apparatus according to claim 1, wherein the conversion unit converts each position in the predetermined direction based on a type of an object existing in the three-dimensional data.   For the position data in the predetermined direction of the three-dimensional data, the conversion means compares position data in a range corresponding to a space where no object exists with position data in a range corresponding to a space where the object exists. The image processing apparatus according to claim 1, wherein the conversion is performed with a large compression ratio.   The conversion unit is configured to determine the position data in the predetermined direction of the three-dimensional data, rather than the position data in a range where the distance from the position corresponding to the space where the reference object exists is a first distance. 4. The image processing apparatus according to claim 1, wherein data in a position in a range that is a second distance farther than the first distance is converted with a larger compression rate. 5.   The said conversion means compresses the data of a position so that a compression rate increases monotonously according to the distance from the position corresponding to the space where the object as the reference exists. Image processing device. A first input means for inputting a user's operation for the compression ratio of the data at the position in the predetermined direction;
The said conversion means converts the data of the position of the said predetermined direction of the said three-dimensional data based on the operation input into the said 1st input means. The image processing apparatus described.   In the case where there are a plurality of objects, the conversion means compares the position data in the range corresponding to the space where the plurality of objects do not exist with the position data in the range corresponding to the space where the plurality of objects exist. The image processing apparatus according to claim 1, wherein conversion is performed with a large compression rate.   The converting means converts the position data in a range corresponding to a space where the predetermined object of the plurality of objects exists, and a ratio of the distance from the predetermined position to each of the two objects including the predetermined object. The image processing apparatus according to claim 7, wherein scaling is also performed in a direction perpendicular to the predetermined direction. A decision means for deciding whether or not to zoom in a direction perpendicular to the predetermined direction;
The determination means is a second input means for acquiring a user operation for selecting whether or not to zoom in a direction perpendicular to the predetermined direction, or is perpendicular to the predetermined direction based on the type of the predetermined object. Determining means for determining whether or not to scale in any direction,
In the case where the converting unit is determined by the determining unit so as to scale the image in a direction perpendicular to the predetermined direction, the position data in a range corresponding to a space where the predetermined object is present among the plurality of objects. The image processing apparatus according to claim 8, wherein the image processing apparatus is also scaled in a direction perpendicular to the predetermined direction.   The three-dimensional data includes two-dimensional image data obtained from an image sensor having photoelectric conversion elements arranged two-dimensionally, and data in the depth direction of a subject as position data in the predetermined direction. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized. It further comprises detection means for detecting a specific object from the image data,
The image processing apparatus according to claim 10, wherein the conversion unit determines an object existing in the three-dimensional data based on a detection result by the detection unit.   The conversion means is based on the thickness information of the object detected by the detection means stored in advance in a range in which the object detected from the image data by the detection means exists in the predetermined direction of the three-dimensional data. The image processing apparatus according to claim 11, wherein the image processing apparatus estimates the image. The imaging element receives light beams of a plurality of different pupil regions of the imaging optical system by the plurality of photoelectric conversion elements,
The position data in the predetermined direction of the three-dimensional data is calculated from a pair of image signals corresponding to the different pupil regions. Image processing apparatus.   The output means converts the three-dimensional data converted by the conversion means into a predetermined format for use in generating a three-dimensional structure, and outputs the converted data. The image processing apparatus according to item.   The image processing apparatus according to claim 1, wherein the output unit converts the three-dimensional data into an STL file format and outputs the STL file format. An acquisition step in which the acquisition means acquires three-dimensional data describing a space including the object;
A conversion step in which conversion means converts the position data of the three-dimensional data in a predetermined direction according to the position where the object exists;
An output method, wherein the output means includes an output step of outputting the three-dimensional data in which the position in the predetermined direction is converted.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 15.