JP2015170907A - Scanner system, data processing method of scanner system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently create electronic document data separating objects or electronic document data integrating objects in response to simultaneously imaging a plurality of mounted objects.SOLUTION: A scanner system includes: mounting means for mounting a plurality of objects; a camera which images the plurality of objects mounted in the mounting means and generates a camera image; and projection means for projecting a predetermined measurement pattern or a user interface on the mounting means. In accordance with an instruction to the user interface, a storage form is set which indicates whether the plurality of objects mounted in the mounting means are combined into one page and stored or the objects are separated and stored for the unit of a page. On the basis of the set storage form, processing for conversion into an electronic document in which the plurality of objects are combined into one page and processing for conversion into an electronic document in which the objects are separated are switched and controlled.

Description

  The present invention relates to a scanner system, a data processing method for the scanner system, and a program.

  Conventionally, when a document is scanned and stored as electronic data, there are a line scanner using a line sensor for imaging and a camera scanner using a two-dimensional imaging sensor. In particular, in the case of a camera scanner in which a camera is arranged above the document table and the document is placed on the document table with the document facing upward, a single document can be quickly scanned and the book can be scanned quickly. Thus, a thick original can be easily placed on the document table and scanned. Further, Patent Document 1 discloses a camera scanner that scans a three-dimensional shape by placing a three-dimensional object on a document table as well as a document such as paper or a book. The camera scanner of Patent Literature 1 includes a light projecting unit together with a camera for capturing an image, images a measurement pattern projected from the light projecting unit with the camera, and measures a three-dimensional shape based on the principle of triangulation. Then, the three-dimensional shape of the object placed on the document table is calculated to determine whether it is a flat manuscript, a book, or a three-dimensional object, and shooting is performed in an appropriate shooting mode according to each.

JP 2003-78725 A

  However, in the camera scanner of Patent Document 1, when a plurality of different types of objects are placed on the document table, it is not possible to shoot in an appropriate shooting mode, and a large number of objects are required because each shooting is required. It took time and effort to shoot. In order to place a plurality of data in the same page when collecting data of a plurality of objects into a PDF-like document, the data is once transferred to a personal computer or the like and the individual data is arranged by a PDF editing software. It took time and effort.

The present invention has been made to solve the above-described problems, and an object of the present invention is to obtain electronic document data obtained by separating each object according to simultaneous shooting of a plurality of placed objects, or It is to provide a mechanism that can efficiently create electronic document data in which each object is integrated.

The scanner system of the present invention that achieves the above object has the following configuration.
A mounting unit for mounting a plurality of objects, a camera for capturing a plurality of objects mounted on the mounting unit and generating a camera image, and irradiating the plurality of objects mounted on the mounting unit First distance image generation means for generating a distance image from an infrared image obtained by photographing the infrared pattern to be obtained, projection means for projecting a predetermined measurement pattern or user interface onto the placement means, and the user interface Setting means for setting a storage format indicating whether a plurality of objects placed on the placing means according to the instruction are combined and stored in one page, or each object is separated and stored one page; Based on the storage format set by the setting means, a process for converting a plurality of objects into a page-combined electronic document, and Characterized in that it comprises control means for controlling switching between processing for converting the electronic document to separate the objects, the.

  According to the present invention, electronic document data in which each object is separated or electronic document data in which each object is integrated can be efficiently created by simultaneously photographing a plurality of placed objects.

It is a figure which shows an example of a scanner system. It is a figure which shows the structural example of the camera scanner shown in FIG. It is a block diagram which shows the hardware structural example of a controller part. It is a figure which shows the function structure of the program for control of a camera scanner. It is a figure explaining the process of a distance image acquisition part. It is a flowchart explaining the detail of the process of a gesture recognition part. It is the figure which represented typically the method of the fingertip detection process. It is a flowchart explaining the process of an object detection part. It is a flowchart which shows the process which a planar original image imaging part performs. FIG. 6 is a schematic diagram for explaining processing of a flat document image photographing unit. It is a flowchart which shows the process which a book image imaging part performs. It is a schematic diagram for demonstrating the process of a book image imaging | photography part. It is a flowchart explaining the process which a solid shape measurement part performs. It is a schematic diagram for demonstrating the process of a solid shape measurement part. It is a flowchart which shows the process of a scan application. It is a flowchart which shows the process of a scan application. It is a flowchart which shows the process of a scan application. It is a figure which shows the example of UI screen displayed with a scanner system. It is a figure which shows the example of UI screen displayed with a scanner system. It is a figure which shows the example of UI screen displayed with a scanner system.

Next, the best mode for carrying out the present invention will be described with reference to the drawings.
<Description of system configuration>
[First Embodiment]

  FIG. 1 is a diagram illustrating an example of a scanner system according to the present embodiment. In this example, the camera scanner 101 is connected to the host computer 102 and the printer 103 via a network 104 such as Ethernet (registered trademark).

  In the network configuration of FIG. 1, a scan function for reading an image from the camera scanner 101 and a print function for outputting scan data by the printer 103 can be executed by an instruction from the host computer 102. Further, it is possible to execute a scan function and a print function by direct instructions to the camera scanner 101 without using the host computer 102.

<Configuration of camera scanner>
FIG. 2 is a diagram showing a configuration example of the camera scanner 101 shown in FIG.
As shown in FIG. 2A, the camera scanner 101 includes a controller unit 201, a camera unit 202, an arm unit 203, a short focus projector unit 207, and a distance image sensor unit 208. The controller unit 201 that is the main body of the camera scanner 101, the camera unit 202 for performing imaging, the short focus projector unit 207, and the distance image sensor unit 208 are connected by an arm unit 203. The arm portion 203 can be bent and stretched using a joint.

FIG. 2A also shows a document table 204 on which the camera scanner 101 is installed. The lenses of the camera unit 202 and the distance image sensor unit 208 are directed toward the document table 204, and can read an image in the reading region 205 surrounded by a broken line.
In the example of FIG. 2, the document 206 is placed in the reading area 205 and can be read by the camera scanner 101. A turntable 209 is provided in the document table 204. The turntable 209 can be rotated by an instruction from the controller unit 201, and the angle between an object (object) placed on the turntable 209 and the camera unit 202 can be changed. Here, the object includes a planar object (two-dimensional object) and a three-dimensional object (three-dimensional object).
The camera unit 202 may capture an image with a single resolution, but it is preferable that the camera unit 202 can capture a high-resolution image and a low-resolution image.
Although not shown in FIG. 2, the camera scanner 101 can further include an LCD touch panel 330 and a speaker 340.

FIG. 2B shows a coordinate system in the camera scanner 101.
Shooting data acquired by the camera scanner 101 is three-dimensional data in the distance image sensor coordinate system obtained by the distance image sensor unit 208, and three-dimensional data in the camera coordinate system obtained by the camera unit 202 and the short focus projector unit 207. is there. Here, in the distance image sensor coordinate system and the camera coordinate system, an image plane captured by the RGB camera unit 503 and the camera unit 202 of the distance image sensor unit 208 is defined as an XY plane, and a direction orthogonal to the image plane is defined as a Z direction. Is.
That is, since these three-dimensional data are obtained by independent coordinate systems, it is necessary to be able to handle both in a unified manner. Therefore, an orthogonal coordinate system is defined in which a plane including the document table 204 is an XY plane, and a direction perpendicular to the XY plane is a Z axis.
Then, the respective three-dimensional data obtained in the distance image sensor coordinate system and the camera coordinate system are converted into predetermined arithmetic processing, specifically, three-dimensional data in an orthogonal coordinate system. For example, a three-dimensional point in the distance image sensor coordinate system is converted into a three-dimensional coordinate in the orthogonal coordinate system by the following equation (1).
Formula 1

Here, [Xs, Ys, Zs, 1] represents three-dimensional coordinates in the distance image sensor coordinate system, and [X, Y, Z] represents three-dimensional coordinates in the orthogonal coordinate system after coordinate conversion. [Rs | ts] is a transformation matrix (rotation matrix Rs and translation vector ts) from the distance image sensor coordinate system to the orthogonal coordinate system.
Note that the relative positions of the distance image sensor coordinate system and the camera coordinate system (rotation matrix and translation vector for conversion to the orthogonal coordinate system) with respect to the orthogonal coordinate system are calibrated in advance by a known calibration method. . Hereinafter, when there is no particular notice and it is expressed as a three-dimensional point group, it represents three-dimensional data in an orthogonal coordinate system.

<Hardware configuration of camera scanner controller>
FIG. 3 is a block diagram illustrating a hardware configuration example of the controller unit 201 illustrated in FIG. 2.

  As shown in FIG. 3, the controller unit 201 includes a CPU 302, a RAM 303, a ROM 304, an HDD 305, a network I / F 306, an image processor 307, a camera I / F 308, a display controller 309, and a serial I / F 310 connected to the system bus 301. Audio controller 311 and USB controller 312.

  The CPU 302 is a central processing unit that controls the operation of the entire controller unit 201. The RAM 303 is a volatile memory. A ROM 304 is a non-volatile memory, and stores a startup program for the CPU 302. The HDD 305 is a hard disk drive (HDD) having a larger capacity than the RAM 303. The HDD 305 stores a control program for the camera scanner 101 executed by the controller unit 201.

The CPU 302 executes a startup program stored in the ROM 304 when the power is turned on or the like. This activation program is for reading a control program stored in the HDD 305 and developing it on the RAM 303. When executing the startup program, the CPU 302 executes the control program developed on the RAM 303 and performs control.
Further, the CPU 302 also stores data used for the operation by the control program on the RAM 303 to read / write. Further, various settings necessary for operation by the control program and image data generated by camera input can be stored on the HDD 305 and read / written by the CPU 302. The CPU 302 communicates with other devices on the network 104 via the network I / F 306.

  The image processor 307 reads and processes the image data stored in the RAM 303 and writes it back to the RAM 303. Note that image processing executed by the image processor 307 includes rotation, scaling, color conversion, and the like.

  The camera I / F 308 is connected to the camera unit 202 and the distance image sensor unit 208 and receives image data captured by the camera unit 202 in accordance with an instruction from the CPU 302 and distance image data captured by the distance image sensor unit 208. Obtain and write to RAM 303. In addition, a control command from the CPU 302 is transmitted to the camera unit 202 and the distance image sensor unit 208 to set the camera unit 202 and the distance image sensor unit 208.

  The controller unit 201 can further include at least one of a display controller 309, a serial I / F 310, an audio controller 311, and a USB controller 312.

  A display controller 309 controls display of image data on the display in accordance with an instruction from the CPU 302. Here, the display controller 309 is connected to the short focus projector unit 207 and the LCD touch panel 330.

The serial I / F 310 inputs and outputs serial signals. Here, the serial I / F 310 is connected to the turntable 209, and transmits an instruction of the rotation start / end and rotation angle of the CPU 302 to the turntable 209. In addition, the serial I / F 310 is connected to the LCD touch panel 330, and the CPU 302 acquires the coordinates pressed via the serial I / F 310 when the LCD touch panel 330 is pressed.
The audio controller 311 is connected to the speaker 340, converts audio data into an analog audio signal in accordance with an instruction from the CPU 302, and outputs audio through the speaker 340.

The USB controller 312 controls an external USB device in accordance with an instruction from the CPU 302. Here, the USB controller 312 is connected to an external memory 350 such as a USB memory or an SD card, and reads / writes data from / to the external memory 350. The detailed configuration of the distance image sensor unit 208 will be described later.
<Functional structure of camera scanner control program>
FIG. 4 is a diagram showing a functional configuration 401 of a control program for the camera scanner 101 executed by the CPU 302 shown in FIG.
Note that the control program for the camera scanner 101 is stored in the HDD 305 as described above, and the CPU 302 develops and executes it on the RAM 303 at startup.
The main control unit 402 is the center of control and controls other units in the functional configuration 401.

  The camera image acquisition unit 407 and the distance image acquisition unit 408 are modules that perform image input processing. The camera image acquisition unit 407 acquires image data captured by the camera unit 202 via the camera I / F 308 and stores it in the RAM 303. The distance image acquisition unit 408 acquires the distance image data output from the distance sensor unit 208 via the camera I / F 308 and stores it in the RAM 303. Details of the processing of the distance image acquisition unit 408 will be described later.

  A gesture recognition unit 409 and an object detection unit 410 are modules that analyze image data acquired by the camera image acquisition unit 407 and the distance image acquisition unit 408 to detect and recognize the movement of an object on the document table 204. Details of the processes of the gesture recognition unit 409 and the object detection unit 410 will be described later. Here, the movement of the object includes an instruction operation by a user's hand operating the projected user interface (details will be described later).

  The planar document image photographing unit 411, the book image photographing unit 412, and the three-dimensional shape measuring unit 413 are modules that actually scan an object. The flat original image photographing unit 411 executes suitable arithmetic processing for the flat original, the book image photographing unit 412 for the book, and the three-dimensional shape measuring unit 413 for the three-dimensional object, and outputs data in a format corresponding to each. Details of the processing of these modules will be described later.

The user interface unit 403 includes an object arrangement display unit 414 and an object arrangement correction unit 415. In response to a request from the main control unit 402, the object arrangement display unit 414 generates a GUI component such as a message, a button, a frame indicating the size or position of a document or an object.
Then, the user interface unit 403 requests the display unit 406 to display the generated GUI component. The object arrangement correcting unit 415 receives a request for changing the paper size of the document or the display size or position of the object, and sends the data to the data management unit 405 for storage. The display unit 406 displays the requested GUI component on the short focus projector unit 207 or the LCD touch panel 330 via the display controller 309.
Since the projector unit 207 is installed toward the document table 204, it is possible to project GUI parts on the document table 204. In addition, the user interface unit 403 receives a gesture operation such as a touch recognized by the gesture recognition unit 409, an input operation from the LCD touch panel 330 via the serial I / F 310, and coordinates thereof. Then, the user interface unit 403 determines the operation content (such as a pressed button) by associating the content of the operation screen being drawn with the operation coordinates. By notifying the main control unit 402 of this operation content, the operator's operation is accepted.
The network communication unit 404 communicates with other devices on the network 104 by TCP / IP via the network I / F 306.

The data management unit 405 includes various kinds of work data generated in executing the program included in the functional configuration 401, such as scan data generated by the flat document image photographing unit 411, the book image photographing unit 412, and the three-dimensional shape measuring unit 413. The data, the document generated by the object arrangement correcting unit 415, the size and position information of the object, and the like are stored in a predetermined area on the HDD 305 and managed. In addition, the object data is collectively stored in one document like PDF.
<Description of Distance Image Sensor and Distance Image Acquisition Unit>

  FIG. 3 shows the configuration of the distance image sensor unit 208 corresponding to the distance image generation means of the present embodiment. The distance image sensor unit 208 is a pattern image type distance image sensor using infrared rays. The infrared pattern projection unit 361 irradiates the object with a three-dimensional measurement pattern with infrared rays that are invisible to human eyes. The infrared camera 362 is a camera that captures a three-dimensional measurement pattern (infrared image) projected on an object. The RGB camera 363 is a camera that captures visible light visible to the human eye using RGB signals.

FIG. 5 is a diagram illustrating the processing of the distance image acquisition unit 408 illustrated in FIG. 5A is a flowchart corresponding to the data processing procedure of the distance image acquisition unit 408 shown in FIG. In the following, FIGS. 5B to 5D explain the distance image measurement principle by the pattern projection method. Each step of the flowchart shown in FIG. 5A is realized by the CPU 302 executing a control program (module shown in FIG. 4). Hereinafter, the module shown in FIG. 4 will be mainly described.
When the distance image acquisition unit 408 starts processing, in S501, the infrared pattern projection unit 361 is used to project a three-dimensional measurement pattern 522 using infrared rays onto the object 521 as shown in FIG. In S502, the infrared camera 362 is used to photograph the three-dimensional measurement pattern 522 projected in S501, and an infrared camera image 524 is acquired.

  In step S503, as shown in FIG. 5C, corresponding points between the three-dimensional measurement pattern 522 and the infrared camera image 524 are extracted. For example, one point on the infrared camera image 524 is searched from the three-dimensional measurement pattern 522, and association is performed when the same point is detected. Alternatively, a pattern around the pixel of the infrared camera image 524 may be searched from the three-dimensional measurement pattern 522 and associated with a portion having the highest similarity.

In S504, the distance from the infrared camera 362 is calculated by performing calculation using the principle of triangulation with the straight line connecting the infrared pattern projection unit 361 and the infrared camera 362 as the base line 523.
For pixels that can be correlated in S503, the distance from the infrared camera 362 is calculated and stored as a pixel value, and for pixels that cannot be correlated, an invalid value is stored as the portion where the distance could not be measured. To do. By performing this for all the pixels of the infrared camera image, a distance image in which each pixel has a distance value is generated.

In step S <b> 505, the RGB image 525 of the object is captured using the RGB camera 363. Since the installation positions of the infrared camera 362 and the RGB camera 363 are different, as shown in FIG. 5D, it is necessary to align the two infrared camera images 524 and the infrared camera image 525 that are respectively captured.
Therefore, in S506, the infrared camera image 524 is converted using coordinate system conversion from the coordinate system of the infrared camera 362 to the coordinate system of the RGB camera 363, and the distance image is matched with the coordinate system of the RGB camera image 525.

  It is assumed that the relative positions of the infrared camera 362 and the RGB camera 363 and the respective internal parameters are known by a prior calibration process. In S507, the RGB value of the RGB camera image 525 is stored in each pixel of the distance image subjected to coordinate conversion in S506, thereby generating a distance image having four values of R, G, B, and distance for each pixel. Then, the processing of the distance image acquisition unit 408 ends.

  As described above, the distance image acquired here is based on the distance image sensor coordinate system defined by the RGB camera 363 of the distance image sensor unit 208. Therefore, as described above with reference to FIG. 2B, the distance data obtained as the distance image sensor coordinate system is converted into a set of point groups in the orthogonal coordinate system. (Hereafter, this set of points is called a three-dimensional point group)

  In the present embodiment, as described above, an infrared pattern projection method is employed as the distance image sensor unit 208, but a distance image sensor of another method can also be used. For example, a stereo system that performs stereo stereoscopic vision with two RGB cameras, or a TOF (Time of Flight) system that measures distance by detecting the flight time of laser light may be used.

<Description of gesture recognition unit>
FIG. 6 is a flowchart illustrating details of data processing of the gesture recognition unit 409 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
In FIG. 6A, when the gesture recognition unit 409 starts processing, initialization processing is performed in S601. In step S602, a three-dimensional point group of an object existing on the document table 204 is acquired. In step S603, the user's hand shape and fingertip are detected from the acquired three-dimensional point group. In S604, a gesture is determined from the detected hand shape and fingertip. In S605, the determined gesture is notified to the user interface unit 403, and the process returns to S602 to repeat the gesture recognition process.

Next, details of the data processing in S601 to S604 will be described.
FIG. 6B is a flowchart of the gesture recognition unit initialization process in S601 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
In step S <b> 611, the gesture recognition unit 409 acquires a distance image from the distance image acquisition unit 408. Here, since the object is not placed on the document table 204 at the start of the gesture recognition unit, the plane of the document table 204 is recognized as an initial state. That is, in S612, the widest plane is extracted from the distance image acquired in S611, and the position and normal vector (hereinafter referred to as plane parameters of the document stage 204) are calculated.

FIG. 6C is a flowchart of the three-dimensional point group acquisition process of S602 shown in FIG.
In S621, one frame of the distance image is acquired from the distance image acquisition unit 408, and is converted into a three-dimensional point group by a predetermined calculation process. In step S622, using the plane parameter of the document table 204, the point group on the plane including the document table 204 is removed from the acquired three-dimensional point group. In this way, the three-dimensional point group of the object existing on the document table 204 is acquired, and the three-dimensional point group acquisition process of S602 is ended.

FIG. 6D is a flowchart of the hand shape and fingertip detection process in S603 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4). FIG. 7 is a diagram schematically illustrating the fingertip detection processing method in FIG.
In S631, a three-dimensional point cloud of the hand is obtained by extracting a three-dimensional point cloud of skin color that is higher than a predetermined height from the plane including the document table 204 from the three-dimensional point cloud acquired in S602.
Reference numeral 701 in FIG. 7A represents a three-dimensional point group of the extracted hand. In S632, a two-dimensional image obtained by projecting the extracted three-dimensional point group of the hand onto the plane of the document table 204 is generated, and the outline of the hand is detected.
Reference numeral 702 in FIG. 7A represents a three-dimensional point group projected on the plane of the document table 204. The projection may be performed by projecting the coordinates of the point group using the plane parameters of the document table 204.
Further, as shown in FIG. 7B, if only the value of the xy coordinate is extracted from the projected three-dimensional point group, it can be handled as a two-dimensional image 703 viewed from the z-axis direction. At this time, it is assumed that each point of the three-dimensional point group of the hand corresponds to which coordinate of the two-dimensional image projected on the plane of the document table 204. In S633, for each point on the detected outer shape of the hand, the curvature of the outer shape at that point is calculated, and a point where the calculated curvature is smaller than a predetermined value is detected as a fingertip.
FIG. 7C schematically shows a method of detecting the fingertip from the curvature of the outer shape. Reference numeral 704 denotes a part of a point representing the outer shape of the two-dimensional image 703 projected onto the plane of the document table 204.
Here, it is considered to draw a circle so as to include five adjacent points among the points representing the outer shape such as 704. Circles 705 and 707 are examples thereof. This circle is drawn in order with respect to all the points of the outer shape, and the diameter (for example, 706, 708) is smaller than a predetermined value (the curvature is small), and is used as a fingertip.
In this example, five points are adjacent to each other, but the number is not limited. In addition, the curvature is used here, but the fingertip may be detected by performing elliptic fitting on the outer shape.
In step S634, the number of detected fingertips and the coordinates of each fingertip are calculated, and the hand shape and fingertip detection process ends. At this time, as described above, since the correspondence between each point of the two-dimensional image projected on the document table 204 and each point of the three-dimensional point group of the hand is stored, the three-dimensional coordinates of each fingertip can be obtained. Can do.
This time, a method of detecting a fingertip from an image projected from a three-dimensional point group onto a two-dimensional image has been described, but the image to be detected by the fingertip is not limited to this. For example, the hand region is extracted from the background difference of the distance image or the skin color region of the RGB image, and the fingertip in the hand region is detected by the same method (external curvature calculation, etc.) as described above. Also good. In this case, since the coordinates of the detected fingertip are coordinates on a two-dimensional image such as an RGB image or a distance image, it is necessary to convert the coordinate information into the three-dimensional coordinates of the orthogonal coordinate system using the distance information of the distance image at that coordinate. is there. At this time, the center of the curvature circle used when detecting the fingertip may be used as the fingertip point instead of the point on the outer shape that becomes the fingertip point.

FIG. 6E is a flowchart of the gesture determination process in S604 illustrated in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
In S641, it is determined whether there is one fingertip detected in S603. If the number of fingertips is not one, the process proceeds to S646, determines that there is no gesture, and ends the gesture determination process. If there is one fingertip detected in S641, the process proceeds to S642, and the distance between the detected fingertip and the plane including the document table 204 is calculated. In S643, it is determined whether or not the distance calculated in S642 is equal to or smaller than a minute predetermined value. If S643 is YES, the process proceeds to S644 and it is determined that there is a touch gesture in which the fingertip touches the document table 204.
In S643, if the distance calculated in S642 is not equal to or smaller than the predetermined value, the process proceeds to S645, where it is determined that the fingertip has moved (the gesture is not touched but the fingertip is on the document table 204), and gesture determination processing is performed. . As described above, the gesture determined here and the coordinates thereof are notified to the main control unit 402 in S605.

<Processing of object detection unit>
FIG. 8 is a flowchart for explaining data processing of the object detection unit 410 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
FIG. 8A is a flowchart showing an outline of processing of the object detection unit 410. When the object detection unit 410 starts processing, in step S801, object detection unit initialization processing is performed. In step S802, detection (object placement detection processing) that an object has been placed on the document table 204 is performed. In step S803, the removal of the object on the document table 204 detected in step S902 is detected (object removal detection process). Next, details of the processing of S801, S802, and S803 will be described.

  FIG. 8B is a flowchart showing a detailed procedure of the object detection unit initialization process in S801. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4). When the object detection unit initialization process is started, one frame of the camera image captured by the camera unit 202 is acquired from the camera image acquisition unit 407 in S811. In step S812, the acquired camera image is stored as a document table background camera image (hereinafter, “document table background camera image” refers to the acquired camera image). In S813, the same camera image is stored as the previous frame camera image. In S814, one frame of the distance image is acquired from the distance image acquisition unit 408. In step S815, the acquired distance image is stored as a document table background distance image, and the object detection unit initialization process is terminated (hereinafter referred to as the distance image acquired here when "document table background distance image" is described). ).

FIG. 8C is a flowchart showing a detailed procedure of the object placement detection process in S802 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
When the object placement detection process is started, one frame of camera image is acquired from the camera image acquisition unit 407 in S821. In step S822, the difference between the acquired camera image and the previous frame camera image is calculated, and a difference value obtained by summing the absolute values is calculated. In S823, it is determined whether or not the calculated difference value is equal to or larger than a predetermined value. If the calculated difference value is less than the predetermined value, it is determined that there is no object on the document table 204, the process proceeds to S828, the current camera image is stored as the previous frame camera image, and the process returns to S821 to continue the processing.
If the difference value is equal to or larger than the predetermined value in S823, the process proceeds to S824, and the difference value between the camera image acquired in S821 and the previous frame camera image is calculated in the same manner as in S822. In S825, it is determined whether or not the calculated difference value is equal to or less than a predetermined value. If the difference value calculated in S825 is larger than the predetermined value, it is determined that the object on the document table 204 is moving, the process proceeds to S828, the current camera image is stored as the previous frame camera image, and the process returns to S821. Continue. If the difference value calculated in S825 is less than or equal to the predetermined value, the process proceeds to S826.
In S826, it is determined from the number of times that S825 becomes YES continuously, whether or not the difference value is equal to or smaller than a predetermined value, that is, whether or not the object on the document table 204 has continued for a predetermined number of frames. If it is determined in S826 that the object on the document table 204 is not still in a predetermined number of frames, the process proceeds to S828, where the current camera image is stored as the previous frame camera image, and the process returns to S821 for processing. to continue.
If it is determined in S826 that the object on the document table 204 is stationary for a predetermined number of frames, the process proceeds to S827 to notify the main control unit 402 that the object has been placed, and the object placement detection process is performed. finish.

FIG. 8D is a detailed flowchart of the object removal detection process of S803 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
When the object removal detection process is started, one frame of camera image is acquired from the camera image acquisition unit 407 in S831. In S832, a difference value between the acquired camera image and the document table background camera image is calculated. In step S833, it is determined whether the calculated difference value is equal to or less than a predetermined value. If the difference value calculated in S833 is larger than the predetermined value determined in advance, there is still an object on the document table 204, so the process returns to S831 to continue the processing. If the difference value calculated in S833 is equal to or smaller than a predetermined value determined in advance, the object on the document table 204 has disappeared, so the object removal is notified to the main control unit 402, and the object removal detection process ends.

<Description of Flat Document Image Shooting Unit>
FIG. 9 is a flowchart illustrating processing executed by the flat document image photographing unit 411 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4). FIG. 10 is a schematic diagram for explaining the processing of the planar document image photographing unit 411 shown in FIG.
When the flat document image photographing unit 411 starts the processing, in S801, one frame of the image from the camera unit 202 is obtained via the camera image obtaining unit 407. Here, since the coordinate system of the camera unit 202 does not face the document table 204 as shown in FIG. 2B, the photographed image at this time is an object as shown in FIG. Both 1001 and the document table 204 are distorted.
In step S902, a pixel-by-pixel difference between the document table background camera image and the camera image acquired in step S901 is calculated, and a difference image is generated so that pixels having a difference are black and pixels having no difference are white. Binarize. Therefore, the difference image generated here is an image in which the region of the object 1001 is black (there is a difference), like the difference region 1002 in FIG. In S903, using the difference area 1002, an image of only the target object 1001 is extracted as shown in FIG.
In step S904, gradation correction is performed on the extracted document area image. In S905, the extracted document area image is subjected to projective transformation from the camera coordinate system to the document table 204, and converted to an image 1003 viewed from directly above the document table 204 as shown in FIG. The projective transformation parameters used here can be obtained from the plane parameters calculated in S612 of FIG. 6B and the camera coordinate system in the processing of the gesture recognition unit 409. As shown in FIG. 10D, the image 1003 obtained here may be tilted depending on how the document is placed on the document table 204.
Therefore, in S906, the image 1003 is approximated to a rectangle and then rotated so that the rectangle becomes horizontal, thereby obtaining an image with no inclination like the image 1004 shown in FIG. In step S907, compression and file format conversion are performed on the extracted image 1004 in accordance with a predetermined image format (for example, JPEG, TIFF, PDF, etc.).
In step S908, the generated image data is stored as a file in a predetermined area of the HDD 305 via the data management unit 405, and the processing of the flat original image photographing unit 411 is ended.

<Processing of book image photographing unit>
FIG. 11 is a flowchart for explaining data processing executed by the book image photographing unit 412 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4). FIG. 12 is a schematic diagram for explaining processing of the book image photographing unit 412 shown in FIG.
When the book image photographing unit 412 starts the processing, in S1101, the camera image obtaining unit 407 is used to obtain the camera image from the camera unit 202, and the distance image obtaining unit 408 is used to obtain the distance image from the distance image sensor unit 208. Get frame by frame. An example of the camera image obtained here is shown in FIG.
In FIG. 12A, a camera image 1201 including a document table 204 and a photographing target book 1211 is obtained. FIG. 12B is an example of the distance image obtained here. In FIG. 12B, a color closer to the distance image sensor unit 208 is represented by a darker color, and a distance image 1202 including the distance from the distance image sensor unit 208 to each pixel on the target object 1212 is obtained. ing.
In FIG. 12B, pixels farther from the distance image sensor unit 208 than the document table 204 are shown in white, and the portion of the target object 1212 that is in contact with the document table 204 (target object). In 1212, the right page) is also white.
In S1102, a calculation process for calculating a three-dimensional point group of the book object placed on the document table 204 from the acquired camera image and distance image is performed. In S1103, the book image is subjected to distortion correction processing from the acquired camera image and the three-dimensional point group calculated in S1102, and a two-dimensional book image is generated. Subsequently, the processing of S1102 and S1103 will be described.

FIG. 11B is a flowchart for explaining the book object three-dimensional point group calculation process of S1102 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
When the three-dimensional point group calculation process for the book object is started, in S1111, the pixel-wise difference between the distance image 1202 and the document table background camera image is calculated, and binarization is performed. As shown in FIG. A camera difference image 1203 in which an area 1213 is shown in black is generated.
In S1112, a calculation process for converting the camera difference image 1203 from the camera coordinate system to the distance image sensor coordinate system is performed, and the camera difference image 1204 including the object region 1214 viewed from the distance image sensor unit 208 as illustrated in FIG. Is generated. In step S1113, a pixel-by-pixel difference between the distance image and the document table background distance image is calculated and binarized to generate a distance difference image 1205 in which the object region 1215 is shown in black as shown in FIG. . Here, a portion having the same color as the document table 204 of the photographing target book 1211 that is the target object may not be included in the object region 1213 in the camera difference image 1204 because the difference in pixel values is small.
Further, the portion of the target object 1212 whose height does not change from the document table 204 is not included in the object area 1215 in the distance difference image 1205 because the distance value from the distance sensor unit 208 is small from the document table 204. There is a case.
Therefore, in S1114, the sum of the camera difference image 1204 and the distance difference image 1205 is taken to generate an object area image 1206 shown in FIG. Here, the object area 1216 is an area having a different color or a different height as compared to the document table 204, and only one of the object area 1213 in the camera difference image 1203 and the object area 1215 in the distance difference image 1205 is used. The object region is represented more accurately than the case.
Since the object area image 1206 is a distance image sensor coordinate system, only the object area 1216 in the object area image 1206 can be extracted from the distance image 1202 in S1115. In S1116, the three-dimensional point group 1217 shown in FIG. 12G is generated by performing arithmetic processing for converting the distance image extracted in S1115 into an orthogonal coordinate system. This three-dimensional point group 1217 is a three-dimensional point group of the book object, and the book object three-dimensional point group calculation process is terminated.

FIG. 11C is a flowchart for explaining the book image distortion correction processing in S1103 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
When the book image distortion correction process is started, in S1121, the object region image 1206 is converted from the distance sensor image coordinate system to the camera coordinate system. In step S1122, the object region is extracted from the camera image 1201 using the object region 1216 in the object region image 1206 converted into the camera coordinate system.
In step S1123, the extracted object region image is projectively converted to the document table plane. In S1124, the object region image obtained by projective transformation is approximated to a rectangle, and the book image 1208 shown in FIG. 12H is generated by rotating the rectangle so that the rectangle becomes horizontal. Since the book image 1208 has one of the approximate rectangles parallel to the X axis, the book image 1208 is subjected to distortion correction processing in the X axis direction thereafter.
In S1125, the leftmost point of the book image 1208 is set to P (point P in (h) of FIG. 12). In S1126, the height of the point P (h1 in FIG. 12H) is acquired from the three-dimensional point group 1217 of the book object. In S1127, a point separated by a predetermined distance (x1 in FIG. 12H) from the point P of the book image 1208 in the X-axis direction is defined as Q (point Q in FIG. 12H). In S1128, the height of the point Q (h2 in FIG. 12 (h)) is acquired from the three-dimensional point group 1217. In step S1219, the distance between the point P and the point Q on the book object (l1 in FIG. 12H) is calculated by linear approximation.
Formula 2

  In S1130, the distance between the PQs is corrected by the calculated distance l1, and the pixels are copied to the positions of the points P ′ and Q ′ on the image 1219 in (h) of FIG. In S1131, the processed point Q is set as the point P, and the process returns to S1128 and the same processing is performed, whereby the correction between the point Q and the point R in (h) of FIG. It is assumed that the pixel is a point Q ′ and a point R ′. By repeating this process for all pixels, the image 1219 becomes an image after distortion correction. In S1132, it is determined whether or not the distortion correction processing has been completed for all points. If completed, the book object distortion correction processing is terminated. As described above, it is possible to generate a book image subjected to the distortion correction by performing the processes of S1102 and S1103.

  After generating the book image subjected to the distortion correction, in S1104, gradation correction is performed on the generated book image. In S1105, compression and file format conversion are performed on the generated book image in accordance with a predetermined image format (for example, JPEG, TIFF, PDF, etc.). In step S1106, the generated image data is stored as a file in a predetermined area of the HDD 305 via the data management unit 405, and the processing of the book image photographing unit 412 is ended.

<Description of the three-dimensional shape measurement unit>
FIG. 13 is a flowchart for explaining data processing executed by the three-dimensional shape measurement unit 413 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4). FIG. 14 is a schematic diagram for explaining the processing of the three-dimensional shape measuring unit 413 shown in FIG.
When the three-dimensional shape measurement unit 413 starts processing, in step S1301, a three-dimensional point group measurement process using the camera unit 202 and the projector unit 207 is performed on an object on the turntable 209 provided in the document table 204. .
FIG. 13B is a flowchart of the three-dimensional point group measurement process executed in S1301 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
When the three-dimensional point group measurement process is started, a three-dimensional measurement pattern 1402 is projected from the projector unit 207 to the object 1401 on the turntable 209 shown in FIG. In S1312, one frame of camera image is acquired from the camera unit 202 via the camera image acquisition unit 407. In step S <b> 1313, the distance of each point on the three-dimensional measurement pattern 1402 is measured from the acquired camera image and the positional relationship between the camera unit 202 and the projector unit 207.
The measurement method here is the same as the measurement method described in S503 of FIG. 5 in the processing of the distance image acquisition unit 408.
In S1314, as in the process of the distance image acquisition unit 408, the distance value of each pixel of the camera image is calculated to generate a distance image. In step S1315, a predetermined calculation process (coordinate conversion process) for converting the coordinates of each pixel of the distance image into the orthogonal coordinate system is performed to calculate a three-dimensional point group. In step S1316, the 3D point group included in the document table plane is removed from the calculated 3D point group using the plane parameters of the document table 204.
In step S <b> 1317, a point whose position is greatly deviated from the remaining three-dimensional point group is removed as noise, and a three-dimensional point group 1403 of the object 1401 is generated. In step S1318, the three-dimensional measurement pattern 1402 projected from the projector unit 207 is turned off. In step S1319, a camera image is acquired from the camera unit 202 via the camera image acquisition unit 407, stored as a texture image when viewed from the angle, and the three-dimensional point cloud measurement process is terminated.

When the first three-dimensional point group measurement process of S1301 is performed, in S1302, a rotation instruction is given to the turntable 209 via the serial I / F 310, and the turntable is rotated by a predetermined angle. The smaller the rotation angle is, the higher the final measurement accuracy is. However, the number of times of measurement increases and it takes time, and therefore, an appropriate rotation angle for the apparatus may be determined in advance. In S1303, the three-dimensional point group measurement process of FIG. 13B is performed again.
At this time, as shown in FIG. 14C, the angles of the object 1401 on the turntable 209, the projector unit 207, and the camera unit 202 are changed. Therefore, a three-dimensional point group viewed from a different viewpoint from the three-dimensional point group 1403 obtained in S1301 is obtained, such as a three-dimensional point group 1404 shown in FIG. Here, the three-dimensional point group is data that characterizes the three-dimensional shape of the placed object.
In other words, the 3D point group 1403 includes a 3D point group that is a blind spot region from the camera unit 202 and the projector unit 207 and for which the 3D point group of one object could not be calculated. (Conversely, a 3D point group not included in the 3D point group 1404 is included in the 3D point group 1403).
It should be noted that in blind spot area photographing, it is determined whether the object has been moved from the arrangement position before photographing to another arrangement position, and photographing control is performed so as to cause the camera unit 202 to photograph.
Therefore, a process of superimposing and synthesizing two three-dimensional point groups 1403 and 1404 viewed from different viewpoints is performed. In step S1304, the three-dimensional point group 1404 measured in step S1303 is subjected to photographing control so that the turntable is rotated reversely by an angle rotated with respect to the placement plane from the initial position and is photographed by the camera unit 202. A three-dimensional point group 1405 that roughly matches the position with the point group 1403 is calculated. In step S1305, a three-dimensional point group synthesis process using feature points is performed.
FIG. 13C is a flowchart showing details of the three-dimensional point group synthesis processing in S1305 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
When the three-dimensional point group synthesizing process is started, in S1321, points that become corners are extracted from the two three-dimensional point groups 1403 and 1405 to be synthesized and set as three-dimensional feature points. In S1322, the correspondence between the feature points of the three-dimensional point group 1403 and the feature points of the three-dimensional point group 1405 is calculated, the distances between all the corresponding points are calculated and added, and the position of the three-dimensional point group 1405 is moved. The position where the sum of the distances between the points is minimized is calculated. In step S1323, the three-dimensional point group 1405 is moved to the calculated position and then superimposed on the three-dimensional point group 1403, thereby synthesizing the two three-dimensional point groups 1403 and 1405. In S1324, since there is a portion where the density of the points is increased by the synthesis, downsampling is performed so that the density of the points becomes uniform. In this way, the combined three-dimensional point group 1406 is generated, and the three-dimensional point group combining process is terminated.

When the three-dimensional point cloud composition processing in S1305 is completed, it is determined in S1306 whether the turntable 209 has rotated one turn. If the turntable 209 has not yet rotated once in S1306, the process returns to S1302, further rotates the turntable 209, and then executes S1303 to measure a three-dimensional point group at another angle. Then, the 3D point group 1405 already generated in S1304 and S1305 is combined with the newly measured 3D point group.
In this way, by repeating the processing from S1302 to S1305 until the turntable 209 makes one turn, a three-dimensional point group of the object 1401 can be generated.

If it is determined in S1306 that the turntable 209 has made one turn, the process proceeds to S1307, and a process of calculating a three-dimensional model from the generated three-dimensional point group is performed.
FIG. 13D is a flowchart showing a detailed procedure of the three-dimensional model calculation process in S1307 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4).
When the three-dimensional model calculation process is started, noise removal and smoothing are performed from the three-dimensional point group in S1331. In S1332, the points of the three-dimensional point group are connected by a plane and meshed. In S1333, the texture saved in S1310 is texture-mapped to the plane obtained by meshing. In step S1333, the texture-mapped data is converted into a standard three-dimensional model data format such as VRML or STL, and the three-dimensional model calculation process is terminated.

  When the three-dimensional model calculation process in S1307 ends, the calculated three-dimensional model is stored in a predetermined area on the HDD 305 via the data management unit 405 in S1308, and the data processing of the three-dimensional shape measurement unit 413 ends.

<Description of main control unit>
15A to 15C are flowcharts for explaining the scan application process executed by the main control unit 402 shown in FIG. In particular, (a) of FIG. 15A is a flowchart showing processing of the main control unit 402 shown in FIG. Each step is realized by the CPU 302 executing a corresponding control program (module shown in FIG. 4). Unless otherwise specified, the main control unit of each step is the main control unit 402.
When the main control unit 402 starts processing, in S1501, the storage method and the paper size are selected. (E) of FIG. 15B is a flowchart showing a storage method / paper size selection process.

  In S1541, the storage method selection start screen shown in FIG. 16A (d) is projected onto the document stage 204 via the user interface unit 403. In FIG. 16A (d), there are a message 1631 for prompting selection of a storage format (storage method), a “one page one” button 1632, and a “one page one stage” button 1633. The “one page” button 1632 selects to store the object data one by one on a new page of the document. The “one page and one stage” button 1633 selects that all objects placed on the turntable 209 of the document table 204 are stored in one new page of the document. The stage has the same meaning as the document table. When the user presses any button, the process proceeds to S1542. Here, “one page” means that a plurality of objects are separated and stored on each page.

  In S1542, when the “one page” button 1632 is pressed, the process proceeds to S1543, and when the “one page and one stage” button 1633 is pressed, the process proceeds to S1544. In S1543, FIG. 16A (e), which is a paper size selection screen for processing one page per page, is projected.

  In FIG. 16A (e), a message 1640 is a message prompting the user to select a paper size, and a frame 1634 is a frame indicating the selected paper size. Buttons 1635 to 1639 for specifying the page area are buttons for selecting a paper size.

Similarly, in S1544, (h) in FIG. 16B, which is a paper size selection screen for one page and one stage processing, is projected. In FIG. 16B (h), a message 1640 is a message prompting the user to select a paper size, and a frame 1634 is a frame indicating the selected paper size. Buttons 1636 to 1639 are buttons for selecting a paper size.
Next, the storage method “one page / one” or “one page / one stage (multiple objects are combined and stored)” selected in S 1545 is stored in the RAM 303.
In S1546, when the buttons 1635 to 1638 are pressed by the user, the process proceeds to S1551. If the “other” button 1639 is pressed, the process advances to step S1547.

  In S1547, a processing screen for “others” is projected. In (f) of FIG. 16A, the buttons 1641 to 1643 can select a standard paper size that was not in (e) of FIG. 16A of the first screen. A button 1644 is a button to be pressed when selecting a non-standard paper size. When the user presses the buttons 1641 to 1643, the process proceeds to S1551. If the button 1644 has been pressed, the process advances to step S1549.

In S1549, a screen for setting a non-standard-size paper size shown in (g) of FIG. 16B is projected. A message 1646 prompts the user to set the paper size. The size of the paper frame is set by touching and moving marks 1645 at the four corners of the frame 1634.
If the confirm button 1647 is pressed in S1550, the size of the paper frame is confirmed, and the process proceeds to S1551. In step S1551, the selected paper size is stored in the RAM 303, and the storage method / paper size selection process ends.

In step S1502, the process waits for an object to be scanned to be placed on the document table 204. (B) of FIG. 15A is the detail of the object mounting waiting process of S1502. When the object placement waiting process is started, in step S <b> 1511, the projector unit 207 projects the screen illustrated in FIG. 16A on the document table 204 via the user interface unit 403.
On the screen of FIG. 16A (a), a message 1601 that prompts the user to place an object on the document table 204 is projected. In step S1512, the processing of the object detection unit 410 is activated. The object detection unit 410 starts executing the process described with reference to the flowchart of FIG. In step S1513, an object placement notification from the object detection unit 410 is awaited. When the object detection unit 410 executes the process of S827 in FIG. 8 and notifies the main control unit 402 of the object placement, in S1513 it is determined that there is an object placement notification, and the object placement waiting process ends.

  When the object placement waiting process in S1502 ends, the main control unit 402 subsequently performs a scan execution process in S1503. The flowchart of (d) of FIG. 15A shows the details of the scan execution process of S1503.

When the scan execution process is started, a multi-object position determination process for an object is performed in S1530. The flowchart of (g) of FIG. 15C is the detail of the multiple object position determination process of the target object in S1530.
In S1580, one frame of the distance image is acquired and converted into a three-dimensional point group. This process is the same as S621.

Next, in step S1581, noise is removed from a point whose height is a predetermined value or less from the three-dimensional point group. Specifically, points with a height of 0.1 mm or less and isolated points are removed as noise. A group of adjacent points collected from the remaining point group is referred to as a point group.
(I) of FIG. 16B shows the state of the point group after noise removal. The point cloud group 1651 is a point cloud of a car that is a three-dimensional object. The point cloud group 1652 is a paper point cloud.

In step S1582, the position of the point group that occupies an area of a predetermined value or more from the continuous point group is stored in the RAM 303 as the position of the object. The area of the point group is the area of the circumscribed rectangle projected onto the document table (XY plane). For example, when the area is 1 cm ² or less, it is not regarded as an object as noise. Examples of the circumscribed rectangle are the frame 1660 and the frame 1661 in FIG. 16B (j). The positions of circumscribed rectangles of the individual objects are stored, and the position determination process for the plurality of objects is completed.

In step S1531, the scan start screen illustrated in FIG. 16A (b) is projected onto the document stage 204 via the user interface unit 403. (B) of FIG. 16A is an example when it is determined that the number of objects on the document table is one in the multiple object position determination process of S1530. In FIG. 16A (b), the object 1611 is a scan target object placed by the user. A 2D scan button 1612 for setting shooting conditions is a button for receiving a shooting instruction for a flat document. A book scan button 1613 for setting shooting conditions is a button for accepting a shooting instruction for a book document.
A 3D scan button 1614 for setting shooting conditions is a button for receiving a measurement instruction of a three-dimensional shape. A scan start button 1615 is a button for receiving an instruction to start execution of the selected scan. As described above, the user interface unit 403 detects that one of the buttons has been pressed by the user from the coordinates of the touch gesture notified from the gesture recognition unit 409 and the coordinates at which these buttons are displayed (hereinafter, referred to as the button). The description of detection by the user interface unit is omitted, and is described as “detecting a touch on a button”). The user interface unit 403 can exclusively select each of the 2D scan button 1612, the book scan button 1613, and the 3D scan button 1614. When a touch on any button of the user is detected, the touched button is set in a selected state, and the selection of other buttons is canceled.

  FIG. 16B (j) is an example in which a plurality of objects on the document table are determined in the multiple object position determination process in S1530, the storage method selected by the user is “one page and one stage”, and the paper size is B4. is there. Two objects, a car 1662 whose object is a three-dimensional object, and a printed material 1663 are placed on the document table.

  In the case of the storage method of one page and one stage, information on where to place each object in the page is required at the time of document creation. The position on the XY plane of the paper frame 1667 and each object with respect to the paper frame is acquired in S1530, and a frame 1660 indicating the area of the car 1662 and a frame 1661 indicating the area of the printed material 1663 are projected. If you want to change the arrangement, you can change it by moving the object. The object arrangement correcting unit 415 monitors whether or not the position of the object has been corrected by the user. The determination is made when the scan start button 1666 is pressed. Three types of buttons 1664 indicating scan types are buttons that allow the user to select a scan type for the frame 1660 of the left car 1662. In this case, the user selects 3D scanning. Similarly, the three types of buttons 1665 are buttons for selecting the type of scanning of the printed matter on the right side. In this case, the user selects 2D scanning.

  In S1532, the process waits until a touch on the scan start button 1615 is detected in the example of FIG. 16A (b) and the scan start button 1666 or the position correction button 1668 is detected in the example of FIG. 16B (j). If a touch on the scan start button 1615 or the scan start button 1666 is detected in S1532, the process proceeds to S1533. If the touch of the position correction button 1668 is detected, the position of the object has been changed by the user, so the process returns to S1530 and the process is repeated.

  In S1533, the scan type selected by the user is determined. If the 2D scan button 1612 is selected, the process proceeds to S1534. If the book scan button 1613 is selected, the process proceeds to S1535. If the 3D scan button 1614 is selected, the process proceeds to S1536. In step S1534, the processing of the planar document image photographing unit 411 is executed. In step S1535, the book image photographing unit 412 is processed. In S1536, the processing of the three-dimensional shape measurement unit 413 is executed.

  In step S1537, it is determined whether all the objects on the document table have been scanned. If all the objects have been scanned, the scan execution process ends. If there is an object that has not been scanned yet in the case where a plurality of objects are placed on the document table, the process returns to S1533 to continue the processing.

  By these scan execution processes, the data management unit 405 stores the two-dimensional data or three-dimensional data of all the objects in the RAM 303. When the storage method is one stage per page, the relative position with respect to the original is also stored.

When the scan execution process in S1503 ends, the main control unit 402 subsequently performs an object removal waiting process in S1504. (C) of FIG. 15A is a flowchart of an object removal waiting process. When the object removal waiting process is started, the scan end screen shown in (c) of FIG. 16A is displayed in S1521.
In the scan end screen of (c) of FIG. 16A, a message 1621 for notifying the user that the scan has ended is projected. In step S1522, the process waits for reception of an object removal notification from the object detection unit 410. Here, the object removal notification is made by the object detection unit 410 in S834 in FIG. When there is an object removal notification in S1522, the object removal waiting process is terminated. When the object removal waiting process in S1504 ends, the main control unit 402 advances to S1501.

In S1501, when the button 1622 is pressed, the process proceeds to S1506. If the preset standby time has passed, it is regarded that the scan is continued, the process returns to S1502, the initial screen of FIG. 16A is displayed, and the object placement on the document table 204 is awaited. In this way, when the user wants to scan a plurality of documents, it can be detected that the document on the document table 204 has been replaced, and a plurality of documents can be scanned.
In step S1506, document creation processing is performed. FIG. 15C is a flowchart showing details.
In S1560, the storage method selected by the user is confirmed. In the case of “one page”, the process proceeds to S1561, and in the case of “one page and one stage”, the process proceeds to S1567.
In S1561, the paper size selected by the user is confirmed. If it is automatic, the process proceeds to S1562. Otherwise, the process proceeds to S1563.
In S1562, a new page is created with the same paper size as the scanned data. That is, if the size of the object (the circumscribed rectangle) is A4, the paper size of the new page is set to A4.

In step S1564, data is arranged on the page. If it is PDF (electronic document data), a page dictionary of A4 size is created to create a new page. If the object is paper or a book, a JPEG image that is two-dimensional data is arranged in the page. Similarly, if the object is a three-dimensional object, U3D data that is three-dimensional data is arranged in the page. The position and size of the three-dimensional data that can be seen at the beginning are stored in the three-dimensional annotation dictionary in the PDF as a default view.
Similarly, if the paper size is not automatic but fits into one standard size, in S1563, a new page of the paper size selected by the user is created regardless of the size of the object.

  In step S1565, the data is scaled to the paper size and arranged on the page. For example, if the object is a printed matter of A3 and the paper size selected for the document is B4, the size is reduced from A4 to B4 and stored in the PDF.

In S1566, it is determined whether data of all objects has been processed. If not all have been processed, the process returns to S1561 and continues. If all processing is completed, the document creation process is terminated.
A case where the storage method selected by the user in S1560 is one page and one stage will be described. In S1567, a new page of the paper size selected by the user is created.
In step S1568, one stage is selected.

In S1569, in the case of the storage method of one page and one stage at the time of scanning the object, since the object data is stored for each stage, information on the object data in the stage is taken out. Data of the object is stored in a new PDF page according to the position information in the page.
For example, in the case of (j) in FIG. 16B, the three-dimensional data of the car 1662 on the left side and the JPEG data of the printed matter 1663 on the right side are stored in the PDF page in the paper frame 1667 having the B4 paper size.

  In step S1570, it is determined whether the data in the stage has been processed. If there is still data on the object, the process returns to S1569 to arrange the data in the page. If all the objects have been processed, the process proceeds to S1571.

  In S1571, it is determined whether all stages have been processed. If the stage remains, the process returns to S1568 and continues. If not, the document creation process is terminated.

As described above, in the first embodiment, the user can select whether to scan a flat document, scan a thick book, or perform solid shape measurement. When all three types of scanning modes are not necessary, for example, it may be possible to execute two types of scanning of a flat document and a thick book according to user settings or the like. In that case, display may be performed so that two scans to be executed can be selected.
Specifically, by projecting only the 2D scan button 1612, the book scan button 1613, and the scan start button 1615 in (b) of FIG. 16A, it is possible to accept an input from the user who selects two types of scans. In addition, when only one type of scan mode is required, for example, only a flat document scan or only a book scan may be executed according to a user setting or the like.
In that case, in FIG. 16A (b), only the scan start button 1615 is projected, and the scan may be executed when a touch on the scan start button 1715 is detected without accepting the selection of the scan type by the user. Further, when there is only one type of scan mode as described above, when the placement of an object on the document table 204 is detected, the scan operation screen as shown in FIG. May be executed.

[Second Embodiment]
In the camera scanner having the configuration of the first embodiment, the object arrangement display unit 414 displays a frame indicating the paper size of the document or a frame indicating the area occupied by the object on the document table 204.
In FIG. 16B (j), there are a paper frame 1667, a frame 1660 indicating the area of the object, and a frame 1661.
When the object is a three-dimensional object, the projection of the frame from the projector unit 207 may be applied to the three-dimensional object and the frame may not be projected correctly.
In the first embodiment, the frames are projected and shown as squares, but any method may be used as long as the area can be shown.

  For example, an area surrounded by four straight lines 1700 in FIG. 17A may be shown as a paper frame. Even if the projection is applied to the three-dimensional object, the user can easily know the paper size and the area occupied by the object by projecting the straight line to the full document table.

[Third Embodiment]
In the camera scanner having the configuration of the first embodiment, the object arrangement display unit 414 displays a frame indicating the paper size of the document or a frame indicating the area occupied by the object on the document table 204. In FIG. 16B (j), there are a paper frame 1667, a frame 1660 indicating the area of the object, and a frame 1661.
When the object is a three-dimensional object, the projection of the frame from the projector unit 207 may be applied to the three-dimensional object and the frame may not be projected correctly.
In the first embodiment, the frames are projected and shown as squares, but any method may be used as long as the area can be shown.

  For example, as shown in FIG. 17B, a marker 1710 indicating a region may be provided outside the paper frame to assist. Even if the projection is applied to the three-dimensional object, the user can easily know the paper size and the area occupied by the object by projecting the marker 1710 toward the end of the document table.

[Fourth Embodiment]
In the camera scanner having the configuration of the first embodiment, the object arrangement display unit 414 displays a frame indicating the paper size of the document or a frame indicating the area occupied by the object on the document table 204. In FIG. 16B (j), there are a paper frame 1667, a frame 1660 indicating the area of the object, and a frame 1661.
When the object is a three-dimensional object, the projection of the frame from the projector unit 207 may be applied to the three-dimensional object and the frame may not be projected correctly.
In the first embodiment, the frames are projected and shown as squares, but any method may be used as long as the area can be shown.
For example, a frequently used paper size may be printed in advance on the document table 204 or a printed paper may be laid.
For example, a case where an A4 portrait frame is printed on the document table 204 will be described.
When the paper size selected by the user is A4, since the A4 frame is printed in advance on the document table as shown in FIG. 17C, it can be confirmed regardless of the position of the three-dimensional object or the projector. .
However, in this case, it becomes a problem when the user does not select the A4 size.

  Therefore, in this embodiment, only when the paper size selected by the user is different from the size of the paper frame printed on the document table, a frame indicating the paper size is projected in S1546 of FIG. 15B (e) of the first embodiment. It is characterized by doing.

  Even when a three-dimensional object is projected in this way, a frame of a frequently used paper size, such as A4 size, is printed in advance on the document table, and the paper frame only when a paper size different from A4 is selected by the user. In many cases, even if a three-dimensional object is placed, the user can correctly know the paper size area.

[Fifth Embodiment]
In the first embodiment, the object placement correcting unit 415 reads a position with respect to the paper frame in S1532 of the scan execution process S1503, and creates a document having pages configured in the same positional relationship when creating a document such as PDF. did. The means for determining the arrangement can obtain the same effect even when the scan is not executed.

In the document creation processing of S1506, as shown in FIG. 17D, thumbnail images 1730 of the object data are arranged on the upper part of the document table, and a paper frame 1734 is projected on the lower part of the document table. It may be shown that the data of the object is arranged at the position of the thumbnail image 1732 with respect to the paper frame 1734 by touching the thumbnail like the finger 1731 and dragging it to the position of the finger 1733.
Further, the page area may be adjusted in accordance with recognizing the movement of the marker by the user and recognizing the movement instruction.

  Each process of the present invention can also be realized by executing software (program) acquired via a network or various storage media by a processing device (CPU, processor) such as a personal computer (computer).

  The present invention is not limited to the above embodiment, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these are excluded from the scope of the present invention. is not.

DESCRIPTION OF SYMBOLS 101 Camera scanner 201 Controller part 202 Camera part 204 Document stand 207 Projector 208 Distance image sensor part

Claims (13)

A mounting means for mounting a plurality of objects;
A camera that shoots a plurality of objects placed on the placing means and generates a camera image;
First distance image generation means for generating a distance image from an infrared image obtained by photographing an infrared pattern irradiated to a plurality of objects placed on the placement means;
Projection means for projecting a predetermined measurement pattern or user interface onto the placement means;
A setting for setting a storage format indicating whether a plurality of objects placed on the placing means are combined into one page and stored according to an instruction to the user interface, or each object is separated and stored one page at a time Means,
Control means for switching control between processing for converting a plurality of objects into electronic document data obtained by page composition based on the storage format set by the setting means, and processing for converting each object into electronic document data separated from each other;
A scanner system comprising:   2. The scanner system according to claim 1, further comprising a first acquisition unit configured to acquire a planar document image associated with the object placed on the placement unit from the camera image according to an instruction to the user interface. . The apparatus according to claim 1, further comprising: a second acquisition unit configured to acquire a book image associated with the object placed on the placement unit from the camera image and the distance image according to an instruction to the user interface. Item 3. The scanner system according to Item 2. Second distance image generation means for generating a distance image from a camera image obtained by photographing the predetermined pattern projected by the projection means on the object in accordance with an instruction to the user interface;
Third acquisition means for acquiring a three-dimensional point group for performing a predetermined coordinate conversion process on the distance image generated by the second distance image generation means and specifying the three-dimensional shape of the object placed on the placement means. The scanner system according to any one of claims 1 to 3, further comprising:   5. The scanner system according to claim 1, wherein the projecting unit projects a page area set in the electronic document data onto the placing unit.   5. The scanner system according to claim 1, wherein the projection unit projects a user interface for setting an imaging condition for imaging the placement unit.   5. The projector according to claim 1, wherein the projecting unit projects a marker for specifying a page region in which the planar object and the solid object placed on the placing unit are placed. Scanner system. Recognizing means for analyzing a distance image generated by the first distance image generating means and recognizing an instruction operation by an operator for the user interface;
The scanner system according to claim 7, further comprising: an adjusting unit that adjusts the page area in response to the recognition unit recognizing a movement instruction for the marker projected by the projection unit. Determining means for determining whether there is a blind spot area in which the entire one object cannot be imaged due to the positional relationship with each object placed on the placing means;
When it is determined by the determination means that there is a blind spot area, the imaging control means for photographing the object placed on the placement means by the camera by rotating the placement means relative to the placement plane;
The scanner system according to any one of claims 1 to 4, further comprising: Determination means for determining whether an object is placed on the placement means based on a camera image of the placement means taken by the camera and a camera image stored in advance;
The scanner system according to claim 9, wherein when it is determined that no object is placed on the placement unit, the shooting control unit does not perform shooting of the blind spot area. When it is determined by the determination means that there is a blind spot area, the determination means for determining whether the object has been moved from the placement position of the placement means to another placement position;
When it is determined that the three-dimensional object has been moved from the placement position of the placement means to another placement position, the shooting control means causes the camera to take a picture of the object placed on the placement means. The scanner system according to claim 9, wherein: A placement unit for placing a plurality of objects, a camera for capturing a plurality of objects placed on the placement unit and generating a camera image, and projecting a predetermined measurement pattern or user interface on the placement unit A data processing method for a scanner system, comprising:
A first distance image generating step for generating a distance image from an infrared image obtained by photographing an infrared pattern irradiated on a plurality of objects placed on the placing means;
A projection step in which the projection means projects a predetermined measurement pattern or user interface on the placement means;
A setting for setting a storage format indicating whether a plurality of objects placed on the placing means are combined into one page and stored according to an instruction to the user interface, or each object is separated and stored one page at a time Process,
Based on the storage format set in the setting step, a control step for switching and controlling a process for converting a plurality of objects into electronic document data obtained by combining pages and a process for converting each object into electronic document data separated from each other;
A data processing method for a scanner system comprising:   A program for causing a computer to execute the data processing method of the scanner system according to claim 12.

Images