JP6203070B2 - Scanning system, control method therefor, and program - Google Patents

Description

  The present invention relates to a scan system, a control method therefor, and a program, and more particularly to a technique for converting and storing a document using a camera scanner.

  Conventionally, when a document is scanned and stored as electronic data, there are a line scanner using a line sensor for photographing and a camera scanner using a two-dimensional area sensor. In particular, in the case of a camera scanner in which a camera is arranged above a document table and an original is placed on the document table and is photographed, a single document can be quickly scanned just by placing it. Furthermore, even when a plurality of originals are placed on top of each other, there are some that are photographed with a camera scanner before and after the placement, extract the outline of each document from the difference image, and scan each document (see, for example, Patent Document 1). ).

JP 2007-201948 A

  When a plurality of originals are placed on top of each other, there are cases where a plurality of candidates are detected as the outlines of the originals or the outlines of the originals cannot be detected simply by taking the difference between the captured images before and after the placement.

  In order to solve the above problems, the present invention has the following configuration. That is, a scanning system that acquires an image of an object placed on a placement unit, which is continuously acquired by the image acquisition unit that acquires an image of the placement unit and its vicinity and the image acquisition unit. Detection means for detecting the placement of the object on the placement unit based on the change in the image, extraction means for extracting the region of the object from the image acquired by the image acquisition means, and the extraction means A conversion unit that converts the region of the object extracted in step 1 into a predetermined coordinate system, and a derivation unit that derives an outline of the region of the object converted by the conversion unit. When the object is further placed on the placement section, the extraction means, the conversion means, and the derivation means are detected by the detection means that the new object is placed on the placement section. Sought from each of the previous and next images Deriving a after loading of the contour of the new object using the new object correspondence between feature points included in the area of that.

  According to the present invention, it is possible to appropriately detect the contour of the photographing target included in the photographed image even when the photographing target is overlaid.

The figure which shows the network structure of a camera scanner system. The figure which shows the external appearance of a camera scanner. The hardware block diagram of a camera scanner. FIG. 4 is a diagram illustrating a configuration example and a sequence of a camera scanner control program. The flowchart and explanatory drawing of a process of a distance image acquisition part. The flowchart of the process of a gesture recognition part. The figure for demonstrating the process which a gesture recognition part performs. The flowchart of a process of an object detection part. 5 is a flowchart of processing of a planar document image photographing unit. The figure for demonstrating the process of a plane original image imaging | photography part. The figure for demonstrating the process of an object detection part. The figure for demonstrating the process of a plane original image imaging | photography part. 5 is a flowchart of processing of a main control unit according to the first embodiment. The figure which shows an example of the GUI display screen which a user interface part displays. The figure for demonstrating the standard size of a document area and printing paper. 9 is a flowchart of processing of a planar document image photographing unit according to the second embodiment. 10 is a flowchart of processing of a planar document image photographing unit according to the third embodiment. Explanatory drawing of the process of the plane original image imaging | photography part which concerns on Embodiment 3. FIG. 10 is a flowchart of processing of an object detection unit according to the fourth embodiment. The figure for demonstrating the process of the object detection part which concerns on Embodiment 4. FIG.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Embodiment 1>
FIG. 1 is a diagram illustrating a network configuration of a scan system including a camera scanner 101 according to the present embodiment. In FIG. 1, a camera scanner 101 is connected to a host computer 102 and a printer 103 via a network 104 such as Ethernet (registered trademark). In the network configuration of FIG. 1, a scan function for reading an image from the camera scanner 101 and a print function for outputting scan data by the printer 103 can be executed by an instruction from the host computer 102. Further, it is possible to execute a scan function and a print function by direct instructions to the camera scanner 101 without using the host computer 102.

  The host computer 102 is a general information processing apparatus, and can store an image read by the camera scanner 101 and process the read image. The printer 103 is an image forming apparatus, and includes, for example, a multi-function peripheral (MFP) or a single-function printer.

[Camera scanner configuration]
FIG. 2 is a diagram illustrating a configuration example of the camera scanner 101 according to the present embodiment. As shown in FIG. 2A, the camera scanner 101 includes a controller unit 201, a camera unit 202, an arm unit 203, a projector 207, and a distance image sensor unit 208. A controller unit 201 which is a main body of the camera scanner 101, a camera unit 202 for performing imaging, a projector 207, and a distance image sensor unit 208 are connected by an arm unit 203. The arm portion 203 can be bent and stretched using a joint.

  FIG. 2A also shows a document table 204 on which the camera scanner 101 is installed. The document table 204 is a placement unit on which an object to be scanned (here, a document 206) is placed. The lenses of the camera unit 202 and the distance image sensor unit 208 are directed toward the document table 204, and can read an image in the reading area 205 surrounded by a broken line. Here, the reading area 205 includes the document table 204 and its vicinity. In the example of FIG. 2, since the document 206 is placed in the reading area 205, the document 206 can be read by the camera scanner 101. A turntable 209 is provided in the document table 204. The turntable 209 can be rotated by an instruction from the controller unit 201, and the angle between an object placed on the turntable 209 and the camera unit 202 can be changed.

  The camera unit 202 may capture an image with a single resolution, but it is preferable that the camera unit 202 can capture images by switching a plurality of resolutions such as high-resolution image capturing and low-resolution image capturing. Although not shown in FIG. 2, the camera scanner 101 may further include an LCD touch panel 330 and a speaker 340.

  FIG. 2B shows a coordinate system in the camera scanner 101. In the camera scanner 101, a plurality of coordinate systems such as a camera coordinate system, a distance image sensor coordinate system, and a projector coordinate system are defined for each hardware device. In these, the image plane captured by the RGB camera unit 503 of the camera unit 202 and the distance image sensor unit 208 or the image plane projected by the projector 207 is defined as an XY plane, and a direction orthogonal to the image plane is defined as a Z direction. . Furthermore, in order to be able to handle the three-dimensional data of these independent coordinate systems in a unified manner, a rectangular coordinate having the plane including the document table 204 as the XY plane and the direction perpendicular to the XY plane as the Z axis is used. Define the system.

As an example of transforming each coordinate system, FIG. 2C shows a rectangular coordinate system, a space expressed using a camera coordinate system centered on the camera unit 202, an image plane captured by the camera unit 202, The relationship is shown. The three-dimensional point P [X, Y, Z] in the orthogonal coordinate system can be converted to the three-dimensional point Pc [Xc, Yc, Zc] in the camera coordinate system by the equation (1).
Here, Rc and tc are constituted by external parameters obtained by the posture (rotation) and position (translation) of the camera with respect to the orthogonal coordinate system, and Rc is called a 3 × 3 rotation matrix and tc is called a translation vector.

Conversely, a three-dimensional point defined in the camera coordinate system can be converted into an orthogonal coordinate system by equation (2).

Further, the two-dimensional camera image plane photographed by the camera unit 202 is obtained by converting the three-dimensional information in the three-dimensional space into two-dimensional information by the camera unit 202. In other words, the three-dimensional point Pc [Xc, Yc, Zc] on the camera coordinate system can be converted by perspective projection conversion to the two-dimensional coordinate pc [xp, yp] on the camera image plane according to Expression (3).
Here, A is a 3 × 3 matrix called an internal parameter of the camera and expressed by a focal length and an image center.

  As described above, by using the equations (1) and (3), the three-dimensional point group represented by the orthogonal coordinate system can be converted into the three-dimensional point group coordinates or the camera image plane in the camera coordinate system. It is assumed that the internal parameters of each hardware device and the position / orientation (external parameters) with respect to the orthogonal coordinate system are calibrated in advance by a known calibration method. Hereinafter, when there is no particular notice and it is expressed as “three-dimensional point group”, it represents three-dimensional data in an orthogonal coordinate system.

[Hardware configuration of camera scanner]
FIG. 3 is a diagram illustrating a hardware configuration example of the controller unit 201 which is the main body of the camera scanner 101.

  3 includes a CPU 302, RAM 303, ROM 304, HDD 305, network I / F 306, image processing processor 307, camera I / F 308, display controller 309, serial I / F 310, audio controller 311 and USB controller 312. . These components are connected to each other via a system bus 301 and can communicate with each other.

  The CPU 302 is a central processing unit that controls the operation of the entire controller unit 201. The RAM 303 is a volatile memory. A ROM 304 is a non-volatile memory, and stores a startup program for the CPU 302. The HDD 305 is a storage unit and is a hard disk drive (HDD) having a larger capacity than the RAM 303. The HDD 305 stores a control program for the camera scanner 101 according to the present embodiment, which is executed by the controller unit 201.

  The CPU 302 executes a startup program stored in the ROM 304 when the power is turned on. This activation program is for reading a control program stored in the HDD 305 and developing it on the RAM 303. When the CPU 302 executes the startup program, the CPU 302 subsequently executes the control program developed on the RAM 303 to perform control. In addition, the CPU 302 stores data used for the operation by the control program on the RAM 303 and performs reading and writing. Further, various settings necessary for the operation by the control program and image data generated by camera input can be stored on the HDD 305 and read / written by the CPU 302. The CPU 302 communicates with other devices on the network 104 via the network I / F 306. The communication method here may be either wired or wireless.

  The image processor 307 reads and processes the image data stored in the RAM 303 and writes it back to the RAM 303. Note that image processing executed by the image processing processor 307 includes processing such as rotation, scaling, and color conversion on an image.

  The camera I / F 308 is connected to the camera unit 202 and the distance image sensor unit 208, acquires image data from the camera unit 202 in accordance with an instruction from the CPU 302, and acquires distance image data from the distance image sensor unit 208 and writes it to the RAM 303. . In addition, a control command from the CPU 302 is transmitted to the camera unit 202 and the distance image sensor unit 208 to set the camera unit 202 and the distance image sensor unit 208.

  The controller unit 201 may further include at least one of a display controller 309, a serial I / F 310, an audio controller 311, and a USB controller 312.

  A display controller 309 controls display of image data on the display in accordance with an instruction from the CPU 302. Here, the display controller 309 is connected to the projector 207 and the LCD touch panel 330.

  The serial I / F 310 inputs and outputs serial signals. Here, the serial I / F 310 is connected to the turntable 209, and transmits an instruction of the rotation start / end and rotation angle of the CPU 302 to the turntable 209. Further, the serial I / F 310 is connected to the LCD touch panel 330, and the CPU 302 acquires the pressed coordinates via the serial I / F 310 when the LCD touch panel 330 is pressed.

  The audio controller 311 is connected to the speaker 340, converts audio data into an analog audio signal in accordance with an instruction from the CPU 302, and outputs audio through the speaker 340.

  The USB controller 312 controls an external USB device in accordance with an instruction from the CPU 302. Here, the USB controller 312 is connected to an external memory 350 such as a USB memory or an SD card, and reads / writes data from / to the external memory 350.

[Functional configuration of camera scanner]
FIG. 4A is a diagram illustrating an example of a functional configuration of the control program 401 of the camera scanner 101 executed by the CPU 302. FIG. 4B is a sequence diagram showing the relationship between the modules of the control program 401.

  The control program 401 of the camera scanner 101 is stored in the HDD 305, and is executed on the RAM 303 when the CPU 302 starts up. The main control unit 402 controls the other modules as shown in FIG. Therefore, each process described later according to the present embodiment is realized by the CPU 302 reading out a program corresponding to each module from the HDD 305 and executing it.

  The image acquisition unit 416 performs image input processing and includes a camera image acquisition unit 407 and a distance image acquisition unit 408. The camera image acquisition unit 407 acquires image data output from the camera unit 202 via the camera I / F 308 and stores the image data in the RAM 303. The distance image acquisition unit 408 acquires the distance image data output from the distance image sensor unit 208 via the camera I / F 308 and stores it in the RAM 303. Details of the distance image and the processing of the distance image acquisition unit 408 will be described later with reference to FIG. Further, the camera unit 202 and the distance image sensor unit 208 can continuously acquire images at regular intervals. In the present specification, one image of the continuous image group is also referred to as a “frame”.

  The recognition processing unit 417 detects and recognizes the movement of the object on the document table 204 from the image data acquired by the camera image acquisition unit 407 and the distance image acquisition unit 408. The recognition processing unit 417 includes a gesture recognition unit 409, an object detection unit 410, a document region extraction unit 419, a document region conversion unit 420, a feature point extraction unit 421, a two-dimensional image document contour calculation unit 422, a distance calculation unit 423, and a document. The contour calculation unit 424 is configured.

  The gesture recognition unit 409 continues to acquire the image on the document table 204 from the image acquisition unit 416, and when detecting an operation instruction by a gesture such as touch, the content is analyzed and notified to the main control unit 402. When the object detection unit 410 receives a notification of object placement waiting processing or object removal waiting processing from the main control unit 402, the object detection unit 410 acquires an image obtained by capturing the document table 204 from the image acquisition unit 416. In addition, the object detection unit 410 detects the timing at which an object is placed on the document table 204 and stops or the timing at which the object is removed. The document area extraction unit 419 extracts a document area from the distance image acquired by the distance image acquisition unit 408.

  The document area conversion unit 420 cuts out the document area extracted by the document area extraction unit 419 from the camera image acquired by the camera image acquisition unit 407 and the distance image acquired by the distance image acquisition unit 408, and performs a document table for the document image. The image information is converted so that the image information is positioned on a plane parallel to 204. The feature point extraction unit 421 extracts feature points from the document of the camera image converted by the document area conversion unit 420. The two-dimensional image document contour calculation unit 422 extracts a contour from the document of the camera image acquired by the document region conversion unit 420. The distance calculation unit 423 calculates the distance between the feature points extracted by the feature point extraction unit 421 and the document outline of the camera image calculated by the two-dimensional image document contour calculation unit 422 and the distance between the feature points. When the object detection unit 410 detects that the document is placed on the document table 204, the document contour calculation unit 424 calculates the distance between the feature points again by the distance calculation unit 423, and calculates the contour of the document. Details of the processing of the recognition processing unit 417 will be described later with reference to FIGS.

  The scan processing unit 418 is a module that actually scans an object, and includes a flat document image photographing unit 411, a book image photographing unit 412, and a three-dimensional shape measuring unit 413 according to the type of scan object. The flat document image photographing unit 411 executes processing suitable for a flat manuscript, the book image photographing unit 412 performs for a book, and the three-dimensional shape measurement unit 413 performs three-dimensional processing, and outputs data in a format corresponding to each processing. . In the present embodiment, since the scan target is a document, only details of the processing of the planar document image photographing unit 411 will be described later with reference to FIGS. The basic operation is the same when scanning other objects.

  The user interface unit 403 includes a GUI part generation display unit 414 and a projection area detection unit 415. The GUI component generation / display unit 414 receives a request from the main control unit 402 and generates a GUI (Graphical User Interface) component such as a message or a button. Then, the GUI component generation display unit 414 requests the display unit 406 to display the generated GUI component. Note that the display area of the GUI component on the document table 204 is detected by the projection area detection unit 415. The display unit 406 displays the requested GUI component on the projector 207 or the LCD touch panel 330 via the display controller 309. Since the projector 207 is installed toward the document table 204, it is possible to project and project GUI parts on the document table 204. In addition, the user interface unit 403 receives a gesture operation such as a touch recognized by the gesture recognition unit 409, an input operation from the LCD touch panel 330 via the serial I / F 310, and coordinates thereof. Then, the user interface unit 403 determines the operation content (such as a pressed button) by associating the content of the operation screen being drawn with the operation coordinates. The user interface unit 403 receives the operator's operation by notifying the main control unit 402 of the determined operation content.

  The network communication unit 404 communicates with other devices on the network 104 by TCP / IP via the network I / F 306. The data management unit 405 stores and manages various data such as work data generated in the execution of the control program 401 in a predetermined area on the HDD 305. The data stored here is, for example, scan data generated by the flat document image photographing unit 411, the book image photographing unit 412, or the three-dimensional shape measuring unit 413.

[Description of Distance Image Sensor Unit and Distance Image Acquisition Unit]
FIG. 3 shows the configuration of the distance image sensor unit 208. The distance image sensor unit 208 is a pattern image type distance image sensor using infrared rays. The infrared pattern projection unit 361 projects a three-dimensional measurement pattern onto an object using infrared rays that are invisible to human eyes. The infrared camera 362 is a camera that reads a three-dimensional measurement pattern projected onto an object. The RGB camera 363 is a camera that captures visible light visible to the human eye using RGB signals.

  The processing of the distance image acquisition unit 408 will be described with reference to the flowchart of FIG. FIGS. 5B to 5D are diagrams for explaining the principle of distance image measurement by the pattern projection method.

  When the process is started, in S501, the distance image acquisition unit 408 projects a three-dimensional shape measurement pattern 522 using infrared rays onto the object 521 using the infrared pattern projection unit 361 as shown in FIG. In S502, the distance image acquisition unit 408 includes the RGB image 523 obtained by photographing the object using the RGB camera 363, and the infrared image photographed including the three-dimensional shape measurement pattern 522 projected in S501 using the infrared camera 362. A camera image 524 is acquired.

  Here, since the installation positions of the infrared camera 362 and the RGB camera 363 are different as shown in FIG. 2, the two RGB images 523 and the infrared camera image 524 photographed respectively as shown in FIG. 5C are taken. The area is different. Therefore, in S503, the distance image acquisition unit 408 performs processing for matching the infrared camera image 524 with the coordinate system of the RGB image 523 using coordinate system conversion from the coordinate system of the infrared camera 362 to the coordinate system of the RGB camera 363. . It is assumed that the relative positions of the infrared camera 362 and the RGB camera 363 and the respective internal parameters are known by a prior calibration process.

  In S504, the distance image acquisition unit 408 extracts corresponding points between the three-dimensional shape measurement pattern 522 and the infrared camera image 524 that has undergone coordinate conversion in S503, as shown in FIG. 5C. For example, the distance image acquisition unit 408 searches for one point on the infrared camera image 524 from the three-dimensional shape measurement pattern 522, and performs association when the same point is detected. Alternatively, the distance image acquisition unit 408 may search the pattern around the pixel of the infrared camera image 524 from the three-dimensional shape measurement pattern 522 and associate it with the portion having the highest similarity.

  In S505, the distance image acquisition unit 408 calculates the distance from the infrared camera 362 by performing calculation using the principle of triangulation with the straight line connecting the infrared pattern projection unit 361 and the infrared camera 362 as the base line 525. The distance image acquisition unit 408 calculates the distance from the infrared camera 362 and stores it as a pixel value for the pixels that can be associated in S504, and cannot measure the distance for the pixels that cannot be associated. Save the invalid value as a part. This is performed for all the pixels of the infrared camera image 524 that have undergone coordinate transformation in S503, thereby generating a distance image in which each pixel has a distance value.

  In S506, the distance image acquisition unit 408 generates a distance image having four values of R, G, B, and distance per pixel by storing the RGB values of the RGB image 523 in each pixel of the distance image. . The distance image acquired here is based on the distance image sensor coordinate system defined by the RGB camera 363 of the distance image sensor unit 208. In step S507, the distance image acquisition unit 408 converts the distance data obtained as the distance image sensor coordinate system into a three-dimensional point group in the orthogonal coordinate system, as described above with reference to FIG. Hereinafter, when there is no particular designation and “3D point group” is described, it indicates a 3D point group in an orthogonal coordinate system.

  In this embodiment, as described above, an infrared pattern projection method is employed as the distance image sensor unit 208, but a distance image sensor of another method may be used. For example, a stereo system that performs stereo stereoscopic viewing with two RGB cameras, or a TOF (Time of Flight) system that measures distance by detecting the flight time of laser light may be used.

[Description of gesture recognition unit]
Details of the processing of the gesture recognition unit 409 will be described with reference to the flowchart of FIG. When the process is started, in S601, the gesture recognition unit 409 performs an initialization process. In the initialization process, the gesture recognition unit 409 acquires one frame of the distance image from the distance image acquisition unit 408. Here, since the object is not placed on the document table 204 at the start of the gesture recognition unit 409, the plane of the document table 204 is recognized as an initial state from the acquired distance image. That is, the gesture recognition unit 409 extracts the widest plane from the acquired distance image, calculates its position and normal vector (hereinafter referred to as plane parameters of the document table 204), and stores them in the RAM 303.

  In step S602, the gesture recognition unit 409 acquires a three-dimensional point group of an object existing on the document table 204 shown in steps S621 to S622. At this time, in S621, the gesture recognition unit 409 acquires one frame of the distance image and the three-dimensional point group from the distance image acquisition unit 408. In step S <b> 622, the gesture recognition unit 409 uses the plane parameter of the document table 204 to remove the point group located in the plane area including the document table 204 from the acquired three-dimensional point group.

  In S603, the gesture recognition unit 409 performs processing for detecting the shape and fingertip of the user's hand from the acquired three-dimensional point group shown in S631 to S634. Here, the method of the fingertip detection process shown in FIG. 7 will be described with reference to the schematic diagram. In S631, the gesture recognizing unit 409 extracts the skin-colored three-dimensional point group within a predetermined height in the plane area including the document table 204 from the three-dimensional point group acquired in S602. To obtain a three-dimensional point cloud. The predetermined height here can be arbitrarily set according to the sensitivity of gesture recognition. A three-dimensional point group 701 shown in FIG. 7A represents a three-dimensional point group for the extracted hand. In S632, the gesture recognition unit 409 generates a two-dimensional image obtained by projecting the extracted three-dimensional point cloud for the hand onto the plane of the document table 204, and detects the outer shape of the hand. A three-dimensional point group 702 shown in FIG. 7A represents a three-dimensional point group when the three-dimensional point group 701 is projected onto the plane of the document table 204. In the projection, each coordinate of the point group can be projected using the plane parameter of the document table 204. Further, as shown in FIG. 7B, if only the value of the xy coordinate is extracted from the value of the three-dimensional point group for the projected hand, it can be handled as a two-dimensional image 703 viewed from the z-axis direction. At this time, the gesture recognition unit 409 stores, in the HDD 305 or the like, information indicating which of the coordinates of the two-dimensional image projected on the plane of the document table 204 corresponds to each point of the three-dimensional point group for the hand. I shall keep it.

  In S633, the gesture recognition unit 409 calculates the curvature of the outer shape at each point on the detected outer shape of the hand, and detects a point where the calculated curvature is larger than a predetermined value as a fingertip. Here, it is assumed that the predetermined value for the curvature is defined in advance and held in the HDD 305 or the like. FIG. 7C schematically shows a method of detecting the fingertip from the curvature of the outer shape. A point 704 represents a part of a point representing the outer shape of the two-dimensional image 703 projected onto the plane of the document table 204. Here, it is considered that a circle is drawn so as to include five adjacent points among points representing the outer shape such as the point 704. Circles 705 and 707 are examples thereof. This circle is drawn in order with respect to all points of the outer shape, and the diameter (for example, the diameter 706 and the diameter 708) is smaller than a predetermined value (the curvature is large), and is used as a fingertip. In this example, five points are adjacent to each other, but the number is not limited. In addition, the curvature is used here, but as another method, the fingertip may be detected by performing elliptic fitting on the outer shape.

  In S634, the gesture recognition unit 409 calculates the number of detected fingertips and the coordinates of each fingertip. At this time, as described above, since information on the correspondence between each point of the two-dimensional image projected on the document table 204 and each point of the three-dimensional point group with respect to the hand is stored, the three-dimensional coordinates of each fingertip are stored. Can be obtained.

  In this example, the method for detecting a fingertip from an image projected from a three-dimensional point group onto a two-dimensional image has been described, but the image to be detected by the fingertip is not limited to this. For example, the hand region is extracted from the background difference of the distance image or the skin color region of the RGB image, and the fingertip in the hand region is detected by the same method (external curvature calculation, etc.) as described above. Also good. In this case, since the coordinates of the detected fingertip are coordinates on a two-dimensional image such as an RGB image or a distance image, it is necessary to convert the coordinate information into the three-dimensional coordinates of the orthogonal coordinate system using the distance information of the distance image at that coordinate. is there. At this time, the center of the curvature circle used when detecting the fingertip may be used as the fingertip point instead of the point on the outer shape that becomes the fingertip point.

  In S604, the gesture recognition unit 409 performs a gesture determination process from the detected hand shape and fingertip shown in S641 to S646. In S641, the gesture recognition unit 409 determines whether or not there is one fingertip detected in S603. If the number of detected fingertips is not one (NO in S641), the process proceeds to S646, and the gesture recognition unit 409 determines that “no gesture”. If the number of detected fingertips is one (YES in S641), the process proceeds to S642, and gesture recognition unit 409 calculates the distance between the detected fingertip and the plane including document stage 204.

  In S643, the gesture recognition unit 409 determines whether or not the distance calculated in S642 is equal to or less than a predetermined value. The predetermined value here is a threshold value for determining whether the fingertip touches the document table 204 or a movement instruction operation, and is a value based on the movement of the fingertip. If it is equal to or smaller than the predetermined value (YES in S643), the process proceeds to S644, and the gesture recognition unit 409 determines that “touch gesture is present”, assuming that the fingertip touches the document table 204. If the calculated distance is not less than or equal to the predetermined value (NO in S643), the process proceeds to S645, and the gesture recognition unit 409 determines that “the fingertip movement gesture is present”. The fingertip movement here is, for example, a gesture in which the fingertip is present on the document table 204 without touching.

  In S605, the gesture recognition unit 409 notifies the determined gesture to the main control unit 402, and then returns to S602 to repeat the gesture recognition process.

  As a result, an accepting unit that accepts an operation instruction by a gesture made in the vicinity of the document table 204 is realized.

[Processing of object detection unit]
The processing of the object detection unit 410 will be described using the flowchart of FIG. When the process is started, the object detection unit 410 performs an initialization process shown in S811 to S813 in S801 of FIG. In step S811, the object detection unit 410 acquires one frame of camera image from the camera image acquisition unit 407 and acquires one frame of distance image from the distance image acquisition unit 408. In S812, object detection unit 410 saves the acquired camera image as the previous frame camera image. In step S813, the object detection unit 410 stores the acquired camera image and distance image as a document table background camera image and a document table background distance image, respectively. In the following description, “document table background camera image” and “document table background distance image” refer to the camera image and distance image acquired here. That is, the initial state of the document table 204 is recognized.

  In step S802, the object detection unit 410 performs detection (object placement detection processing) that an object has been placed on the document table 204. Details of the processing will be described later with reference to FIGS. 8B and 8C.

  In step S803, the object detection unit 410 detects that the object on the document table 204 that has been placed in step S802 is removed (object removal detection processing). Details of the processing will be described later with reference to FIG.

(Object placement detection process)
FIG. 8B is a flowchart showing details of the object placement detection process in S802, which is a document placement detection process for the first document. This processing is performed after the initialization processing in FIG. 8A and after the object removal detection processing in FIG. 8D. Here, it is assumed that the image acquisition unit 416 continuously captures image groups at a predetermined interval.

  In step S821, the object detection unit 410 acquires one frame of the camera image and the distance image from the continuous image group acquired by the image acquisition unit 416. In S822, the object detection unit 410 calculates a difference between the camera image acquired in S821 and the previous frame camera image already held, and calculates the sum of absolute values as a difference value. In S823, object detection unit 410 determines whether or not the calculated difference value is equal to or greater than a predetermined value. Here, it is assumed that the predetermined value is defined in advance and held in the HDD 305 or the like.

  If the calculated difference value is less than the predetermined value (NO in S823), object detection unit 410 determines that there is no object on document table 204, proceeds to S828, and object detection unit 410 The camera image and distance image of the frame are stored as the previous frame camera image and distance image. Thereafter, the process returns to S821 to continue the processing. If the difference value is equal to or greater than the predetermined value (YES in S823), the process proceeds to S824, and the object detection unit 410 calculates the difference value between the camera image acquired in S821 and the previous frame camera image in the same manner as S822.

  In S825, object detection unit 410 determines whether or not the calculated difference value is equal to or smaller than a predetermined value. Here, it is assumed that the predetermined value is defined in advance and held in the HDD 305 or the like. Note that the same value as the predetermined value used in S823 may be used, or a different value may be used. If the calculated difference value is larger than the predetermined value (NO in S825), object detection unit 410 determines that “the object on document table 204 is moving”, and proceeds to S828, where the camera image of the current frame is displayed. Save as previous frame camera image. Thereafter, the process returns to S821 to continue the processing. If the calculated difference value is equal to or smaller than the predetermined value (YES in S825), the process proceeds to S826. At this time, the object detection unit 410 counts the number of times that the calculated difference value is continuously determined to be equal to or less than a predetermined value.

  In S826, the object detection unit 410 determines whether or not the number of times continuously determined to be equal to or less than the predetermined value in S825 is equal to or greater than the threshold (that is, the predetermined frame indicates that the object on the document table 204 is stationary). It is determined whether or not the number has continued. If the state on which the object on the document table 204 is stationary does not continue for the predetermined number of frames (NO in S826), the process proceeds to S828. In S828, object detection unit 410 saves the current frame camera image and distance image as the previous frame camera image and distance image. Thereafter, the process returns to S821 to continue the processing.

  If the object on the document table 204 is stationary for a predetermined number of frames (YES in S826), the process proceeds to S827, and the object detection unit 410 notifies the main control unit 402 that the object has been placed. Notice. Then, the object placement detection process ends.

  FIG. 8C is a flowchart showing details of the object placement detection process in S802, which is a document placement detection process for the second and subsequent documents. The same processes as those in FIG. 8B are denoted by the same reference numerals and description thereof is omitted.

  In step S831, the object detection unit 410 calculates a difference between the acquired distance image and the previous frame distance image and sums the absolute values. In the process of the first frame for the second document, the image obtained in S828 of FIG. 8B is used as the previous frame distance image.

  In S832, object detection unit 410 determines whether or not the calculated difference value is equal to or greater than a predetermined value. Here, it is assumed that the predetermined value is defined in advance and held in the HDD 305 or the like. If the calculated difference value is less than the predetermined value (NO in S832), object detection unit 410 determines that a new document (undetected document) is not captured, and proceeds to S828. If the difference value is equal to or greater than the predetermined value (YES in S832), object detection unit 410 determines that a new document is captured, proceeds to S833, and performs document contour calculation processing. Details of the processing will be described later with reference to FIGS.

(Object removal detection process)
FIG. 8D is a detailed flowchart of the object removal detection process in S803. When the object removal detection process is started, the object detection unit 410 acquires one frame of the camera image from the camera image acquisition unit 407 in S841. In S842, object detection unit 410 calculates a difference value between the acquired camera image and the document table background camera image. In S843, object detection unit 410 determines whether or not the calculated difference value is equal to or less than a predetermined value. Here, it is assumed that the predetermined value is defined in advance and held in the HDD 305 or the like. Note that the same value as that used in S823 in FIG. 8B may be used.

  If the calculated difference value is larger than the predetermined value (NO in S843), there is still an object on the document table 204, so the process returns to S841 to continue the processing. If the calculated difference value is equal to or smaller than the predetermined value (YES in S843), object detection unit 410 determines that there is no object on document table 204, and notifies main control unit 402 of object removal. Thereafter, the object removal detection process is terminated.

(Document contour calculation processing)
FIG. 8E is a flowchart showing details of the document contour calculation processing in S833. FIG. 11 is a schematic diagram for explaining the document contour calculation processing. In the present embodiment, only the first frame in which the distance between the area indicating the plane of the document table 204 and the document area obtained from the distance image is within a predetermined value determined by the depth of field of the camera unit 202 is recorded. Contour calculation processing is performed. That is, the outline of the document is calculated only for the image at the time when the document is placed on the document table 204.

  When the document contour calculation process is started, the document contour calculation unit 424 uses the document region extraction unit 419 to extract a document region from the distance image in S851. FIG. 11A shows an example of a camera image in the process of placing the second original on the document table 204. FIG. 11B is an example of a distance image in the process of placing the second original on the document table 204. In FIG. 11B, as the distance from the distance image sensor unit 208 increases, the position value in the distance image becomes darker. FIG. 11C shows a difference image from the previous frame distance image. Here, an area having a difference between the previous frame distance image and the current frame distance image is determined as the document area. Of course, there may be a hand holding the document in the distance image, but the region corresponding to the hand can be detected and separated from the document region by, for example, the processing in S603 of the gesture recognition unit 409 described above. .

  In step S852, the document contour calculation unit 424 determines whether the distance between the document area and the document table 204 has become a predetermined value or less. This predetermined value is a threshold value indicating whether or not an original is already placed on the document table 204 and the second and subsequent originals are approaching each other in an overlapping manner. This predetermined value is defined in advance and held in the HDD 305 or the like. In other words, if it is not less than the predetermined value (NO in S852), document outline calculation unit 424 determines that the document is not placed on document table 204, and ends the process without performing the document outline calculation process. If it is equal to or smaller than the predetermined value (YES in S852), the process proceeds to S853, and the document contour calculation unit 424 determines whether or not it is the first frame in which YES is determined in S852. That is, it is determined whether or not it has already been detected in the previous frame that the corresponding document has approached the document table 204 by a predetermined distance or more. If it is not the first frame (NO in S853), the document contour calculation unit 424 ends the process without performing the document contour calculation process.

  If it is the first frame (YES in S853), the process advances to S854, where the document contour calculation unit 424 extracts a document region from the camera image and the distance image and arranges the document region in association with a plane parallel to the document table 204. Specifically, since the document area can be cut out from the distance image in S851, the document outline calculation unit 424 converts the camera image into the distance image coordinate system, and cuts out the document area from the camera image. FIG. 11D and FIG. 11E show the document areas of the cut-out camera image and distance image, respectively. The document contour calculation unit 424 arranges these images on a plane parallel to the document table 204. That is, the document contour calculation unit 424 converts the document into the orthogonal coordinate system of the document table 204.

  In the examples of FIGS. 11D and 11E, the conversion is a two-dimensional distortion. Generally, since a document held in a hand is distorted three-dimensionally, it is necessary to adopt a method of converting the three-dimensional distortion. There is. As a conversion method, for example, “Estimation of non-rigid body deformation using a developable surface model” (Takashi Nakajima et al., “Estimation of non-rigid body deformation using a developable surface model”, July 22, 2009, No. 12 It is possible to develop a flat image using the symposium on recognition and understanding of images (MIRU 2009). FIGS. 11 (f) and 11 (g) show images obtained by converting the document area cut out from the camera image and the distance image into the orthogonal coordinate system of the document table 204, respectively. In this example, the Z-axis direction is flattened in accordance with the position closest to the document table 204 in the document area of the distance image (the value in the Z-axis direction at this time is Z0).

  In step S855, the document contour calculation unit 424 extracts feature points from the document region of the converted camera image. As a feature point extraction method, various methods such as SIFT (Scale-invariant feature transform), which is a feature point calculation method having relatively strong resistance to change, rotation, and enlargement / reduction of illumination, have been proposed. Detailed explanation is omitted. FIG. 11H is an example in which feature points are extracted from the image of FIG. In FIG. 11H, the feature points are represented by black points (●), and a total of 24 feature points such as feature points 1103 and 1104 are extracted. The positions of the feature points 1103 and 1104 in the rectangular coordinate system of the document table 204 are (X1103, Y1103, Z0) and (X1104, Y1104, Z0), respectively.

  In step S856, the document contour calculation unit 424 extracts the document contour from the converted camera image. This is the outermost contour of the camera image converted by the processing of S854, and is as shown in FIG. In step S857, the document contour calculation unit 424 calculates the distance between the feature points extracted in step S855 and the contour extracted in step S856. In the example of FIG. 11 (j), the distance between the feature points 1103 and 1104 is D1. The distances between the feature point 1103 and the left side, right side, top side, and bottom side of the outline of the document are D2, D3, D4, and D5, respectively. Assume that the coordinates of the intersection of the feature point 1103 and the document outline are P2, P3, P4, and P5, respectively. The distances between the feature point 1104 and the left side, right side, top side, and bottom side of the outline of the document are D6, D7, D8, and D9, respectively. The coordinates of the intersection of the feature point 1104 and the outline of the document are P6, P7, P8, and P9, respectively. The document contour calculation unit 424 calculates the distance for other feature points, derives the relationship between the feature points, and ends the document contour calculation process.

[Description of flat document image capture unit]
The processing executed by the flat document image photographing unit 411 will be described with reference to the flowcharts of FIGS. FIG. 9A is a flowchart of processing executed by the flat document image photographing unit 411 for the first document, and FIG. 10 is a schematic diagram for explaining the processing.

  When the process is started, the planar document image photographing unit 411 acquires one frame of the image from the camera unit 202 via the camera image acquisition unit 407 in S901. Here, the coordinate system of the camera unit 202 does not face the document table 204 as shown in FIG. For this reason, the photographed image at this time is distorted in both the object 1001 and the document table 204 as shown in FIG.

  In step S902, the planar document image photographing unit 411 calculates a pixel-by-pixel difference between the document table background camera image and the camera image acquired in step S901, generates a difference image, and sets the difference pixel to black. Binarization is performed so that pixels with no white become white. As a result, the difference image generated here is an image in which the area of the object 1001 is black (there is a difference), like the difference area 1002 in FIG.

  In step S903, the planar document image photographing unit 411 uses the difference area 1002 to extract an image of only the object 1001 as illustrated in FIG. In step S904, the planar document image photographing unit 411 performs gradation correction on the extracted document area image. In step S905, the planar document image photographing unit 411 performs projective transformation from the camera coordinate system to the document table 204 with respect to the extracted document region image, and the image is viewed from directly above the document table 204 as shown in FIG. Converted to the image 1003. The projective transformation parameters used here can be obtained from the plane parameters calculated in S601 of FIG. 6 and the camera coordinate system in the processing of the gesture recognition unit 409.

  Note that, as shown in FIG. 10D, the image 1003 obtained here may be inclined depending on how the document is placed on the document table 204. In step S906, the flat document image photographing unit 411 approximates the image 1003 to a rectangle and then rotates the rectangle so that the rectangle becomes horizontal, and there is no inclination as in the image 1004 shown in FIG. Get an image. As shown in FIG. 10F, rectangular inclinations θ1 and θ2 with respect to the reference line are calculated, and the smaller inclination (here, θ1) is determined as the rotation angle of the image 1003. Alternatively, as shown in FIG. 10G and FIG. 10H, the OCR process is performed on the character string included in the image 1003, the rotation angle of the image 1003 is calculated from the inclination of the character string, and the top / bottom determination process. You may do.

  In step S907, the planar document image photographing unit 411 performs compression and file format conversion on the extracted image 1004 in accordance with a predetermined image format. Examples of the image format include JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and PDF (Portable Document Format). Then, the flat document image photographing unit 411 stores the file as a file in a predetermined area of the HDD 305 via the data management unit 405, and ends the process.

  FIG. 9B is a flowchart of processing executed by the flat original image photographing unit 411 for the second and subsequent originals, and FIG. 12 is a schematic diagram for explaining the processing. The same processes as those in FIG. 9A are denoted by the same reference numerals and description thereof is omitted.

  In step S <b> 911, the planar document image photographing unit 411 performs projective transformation on the acquired camera image (FIG. 12A) to the orthogonal coordinate system of the document table 204. As a result, an image as shown in FIG. 12B is obtained. In step S912, the planar document image photographing unit 411 extracts feature points of the camera image subjected to projective transformation. As the feature point extraction method, the same method as that used in S853 of FIG. FIG. 12C is an example in which feature points are extracted from the image of FIG. In FIG. 12C, the feature points are represented by black points (●), and a total of 27 feature points such as feature points 1203 and 1204 are extracted. The positions of the feature points 1203 and 1204 in the rectangular coordinate system of the document table 204 are (X1203, Y1203, 0) and (X1204, Y1204, 0), respectively.

  In step S913, the planar document image photographing unit 411 compares the feature point obtained in step S855 with the feature point obtained in step S912, and leaves a matching feature point. Even if they do not match completely, they may be considered to be matched if the degree of matching is not less than a predetermined value. In FIG. 12D, it is assumed that 24 feature points match and the feature points 1203 and 1204 match the feature points 1103 and 1104 in FIG. 11H, respectively.

  In step S914, the planar document image photographing unit 411 calculates the distance between feature points. In the example of FIG. 12E, the distance between the feature points 1203 and 1204 is D1 '. The distance is also calculated for other feature points. In S915, planar document image photographing unit 411 calculates the contour of the document. Since the distance between the feature points calculated in S857 and the distance between the feature points and the contour and the distance between the feature points calculated in S914 and the distance between the feature points and the contour are similar to each other, The distance to the contour can be derived. Further, since the coordinates of the intersection of the feature point calculated in S857 and the outline of the document and the coordinates of the feature point calculated in S914 and the coordinates of the intersection of the feature point and the outline of the document are similar, Intersections between feature points and document outlines can also be derived. Thus, the contours of the second and subsequent documents can be calculated. An example of the calculated document outline is shown in FIG. That is, for the same original, even if the shooting state changes, the relationship between the feature points does not change. Therefore, an area for the document on the side of the document to be placed is extracted from each of the images before and after the second and subsequent documents are placed on the already placed document, and the second and subsequent pages are determined from the correspondence of the feature points. The outline after placing the original is specified.

  In step S916, the flat original image photographing unit 411 cuts out the original along the outline of the original calculated in step S915. In step S917, the flat document image photographing unit 411 rotates so that the document cut out in the same manner as in step S906 is horizontal.

[Description of main control unit]
Processing of the scan application executed by the main control unit 402 will be described using the flowchart of FIG.

  When the process is started, in step S1301, the main control unit 402 performs an object placement waiting process in order to detect placement of an object to be scanned on the document table 204. When the object placement waiting process is started, the main control unit 402 projects and displays an initial screen on the document stage 204 by the projector 207 via the GUI part generation / display unit 414 of the user interface unit 403 in S1311. For example, a GUI component of a message 1441 that prompts the user to place an object on the document table 204 as shown in FIG. 14A is generated and displayed.

  In step S1312, the main control unit 402 activates the processing of the object detection unit 410. The object detection unit 410 starts executing the process described with reference to the flowchart of FIG. In step S <b> 1313, the main control unit 402 waits for an object placement notification from the object detection unit 410. When the object detection unit 410 executes the processing of S827 in FIGS. 8B and 8C and notifies the main placement to the main control unit 402, in S1313, the main control unit 402 indicates “There is an object placement notification”. (YES in S1313).

  When the object placement waiting process in S1301 ends, the main control unit 402 displays a GUI component that is a menu display for performing a scanning operation in S1302. For example, the main control unit 402 generates a GUI component including menu buttons (1442 to 1444) for setting a scan processing mode as shown in FIG. Then, the main control unit 402 initially projects the GUI component on the document table 204 by the projector 207 via the GUI component generation / display unit 414.

  After the GUI component update process, in step S1303, the main control unit 402 performs a scan execution process. At the start of the scan execution process, the scan start screen shown in FIG. 14B is projected on the document stage 204 via the GUI component generation display unit 414. The 2D scan button 1442 is a button for accepting an instruction for photographing a flat original. The book scan button 1443 is a button for accepting a book document shooting instruction. The 3D scan button 1444 is a button for receiving a three-dimensional shape measurement instruction.

  As described above, the user interface unit 403 detects that one of the buttons has been pressed by the user from the coordinates of the touch gesture notified from the gesture recognition unit 409 and the coordinates at which these buttons are displayed. Hereinafter, description of detection by the user interface unit 403 is omitted, and “detection of touch on a button” is described. When the user interface unit 403 detects a touch on the 2D scan button 1442, the book scan button 1443, or the 3D scan button 1444, the user interface unit 403 executes the selected scan. Alternatively, a scan start button that receives an instruction to start executing the selected scan may be separately arranged so that each of the 2D scan button 1442, the book scan button 1443, and the 3D scan button 1444 can be exclusively selected. In that case, when detecting the touch of any button of the user, the user interface unit 403 selects the touched button and cancels the selection of the other buttons.

  In step S1331, the main control unit 402 waits until it detects a touch of the menu button (1442 to 1444). If the touched scan start button is the 2D scan button 1442, the process proceeds to S1332, and the main control unit 402 executes the process of the flat document image photographing unit 411. In the case of the book scan button 1443, the process proceeds to S1333, and the main control unit 402 executes the process of the book image photographing unit 412. In the case of the 3D scan button 1444, the process proceeds to S1334, and the main control unit 402 executes the process of the three-dimensional shape measurement unit 413. When the process according to any one of S1332 to S1334 ends, the scan execution process ends.

  After the scan execution processing in S1303, the main control unit 402 performs object removal waiting processing / object placement waiting processing in S1304. When the object removal waiting process / object placement waiting process is started, the GUI component generation display unit 414 displays a scan end screen in S1341. For example, a GUI component of a message 1445 for notifying the user that the scan as shown in FIG.

  In S 1342 and S 1343, the main control unit 402 waits to receive an object placement notification or an object removal notification from the object detection unit 410. The object removal notification is notified by the object detection unit 410 in S844 in FIG. When the object placement notification is received (YES in S1342), the process returns to S1302, and the process is continued. When the object removal notification is received (YES in S1343), the object removal waiting process is terminated.

  After the object removal waiting process / object placement waiting process of S1304, the main control unit 402 proceeds to S1305 and performs a scan end determination process. The scan end determination is made based on a scan end command transmitted from the host computer 102 via the network I / F 306, an end command input from the LCD touch panel 330, or a timer setting (not shown). If a scan end command has been received (YES in S1304), main control unit 402 ends the scan process. When the scanning process is continued (NO in S1304), the process returns to S1301, and the initial screen of FIG. 14A is displayed to wait for the object placement on the document table 204.

  Note that the screen configuration shown in FIG. 14 is merely an example, and other screens and messages may be displayed. As described above, when the user wants to scan a plurality of documents, it can detect that the document on the document table 204 has been replaced, and a plurality of documents can be continuously scanned.

  As described above, when the document is about to be stacked, the information of the document on the side to be stacked is acquired, so that the outline of the document can be appropriately calculated when the document is stacked.

  In the present embodiment, the process for the first original and the process for the second and subsequent originals are described separately, but the processes shown in FIGS. 8C and 9B are performed from the first original. You may do it.

  In the present embodiment, the document contour calculation process is performed only once. However, as another processing method, when the distance between the extracted document area and the document table is within a predetermined range as in S861 of the flowchart of FIG. 8 (f), subsequent processing may be performed. . In that case (YES in S861), document outline calculation unit 424 extracts feature points in a plurality of frames until the document is placed, calculates the distance between feature points and the outline of the document, and calculates the distance between feature points. To do. As a result, distances between different feature points and distances between feature points and contours can be obtained from each frame. For example, the distances between corresponding feature points and distances between feature points and contours are averaged. Then, the feature points after the document is placed on the document table 204 are compared, and the contour of the document can be calculated using the matched feature points.

<Embodiment 2>
In the first embodiment, as shown in FIGS. 9A and 9B, the contour of a flat document placed on the document table 204 is calculated, the document region is cut out, and the document region is rotated so as to be horizontal. Image compression and format conversion. When the stored data is transmitted to the printer 103 and printed, for example, it is assumed that the document area (1501 in FIG. 15) of the stored data is slightly larger than the A4 size (1502 in FIG. 15). In the paper feed cassette (not shown) of the printer 103, papers of two types A3 and A4 are stacked, and when the paper feed setting is set to automatic paper selection, the A3 paper is used although the image size is a little larger. Is selected (1503 in FIG. 15). In view of this, the present embodiment provides a mechanism for correcting the document area to the standard size when the document area of the data to be stored is close to the standard size (A3, A4, etc.) of the printing paper.

  FIGS. 16A and 16B are flowcharts of processing executed by the flat document image photographing unit 411 according to the present embodiment. In this flowchart, the original area and standard size comparison process in S1601, the comparison process with the threshold value in S1602, and the original area reduction process are added to the flowcharts of FIGS. 9A and 9B according to the first embodiment. Has been. The other processes are the same as those of the flat document image photographing unit 411 described with reference to FIG. The standard size is described as having two types of paper sizes A3 and A4. Needless to say, there is no problem even if there are other standard sizes.

  In step S1601, the planar document image photographing unit 411 compares the document area with the standard size. In the example of FIG. 15, the document area 1501 is larger than A4 and smaller than A3. Since it is larger than A4 (YES in S1601), in S1602, the flat document image photographing unit 411 compares the ratio of the document area and the standard size with a threshold value. In the example of FIG. 15, if the size ratio of the document area 1501 and A4 is 103% and the threshold value is 105%, it is smaller than the threshold value. Note that the threshold is defined in advance and held in the HDD 305 or the like. Each threshold value may be set for each type of paper size.

  Since it is smaller than the threshold value (YES in S1602), the planar document image photographing unit 411 reduces the document area to a standard size in S1603.

  As described above, according to the present embodiment, in addition to the effects of the first embodiment, when the document area of the data to be stored becomes a size close to the standard size of the printing paper, the document area can be corrected to the standard size.

<Embodiment 3>
In the first and second embodiments, the camera image and the distance image are used to specify the document area. In the present embodiment, a method is provided in which only the camera image is used and the user is instructed to reposition the document when the document area cannot be specified.

  17A and 17B are flowcharts of processing executed by the flat document image photographing unit 411 according to the present embodiment. In this flowchart, the description of the same processing as in the flowcharts of FIGS. 9A and 9B is omitted. Here, the processing for the first document is the same as that in the first embodiment (FIG. 9A), and therefore the description of FIG. 17A is omitted.

  In FIG. 17B, in S1701, the planar document image photographing unit 411 performs projective transformation on the acquired camera image. FIG. 18A shows a projective conversion of the image in the document table 204 when the first original is placed. FIG. 18B is a result of projective conversion of an image in the document table 204 when the second original is placed on top of each other. FIG. 18C shows a projection conversion of the image in the document table 204 when the third original is placed on top of each other.

  In step S <b> 1702, the planar document image photographing unit 411 generates a difference image between the previous camera image after the projective conversion and the current camera image, and binarizes the difference image. FIG. 18D shows a binarized difference image when the second original is placed on top of each other. That is, the difference image between FIG. 18A, which is the previous camera image, and FIG. 18B, which is the current camera image, is binarized. A portion with a difference becomes a black image. FIG. 18E is a binary image of the difference image when the third original is placed on top of each other. Since there is no difference between FIG. 18B and FIG. 18D, there is no black image region in FIG. Therefore, the difference between the first sheet and the second sheet can be detected, but the difference between the second sheet and the third sheet cannot be detected.

  In step S1703, the planar document image photographing unit 411 extracts a rectangle as a document region. As a rectangle extraction method, a method described in the prior art (for example, Japanese Patent Application Laid-Open No. 2007-201948) can be used. A rectangle extracted from the difference image in FIG. 18D is shown in FIG. In FIG. 18F, two types of rectangles 1801 and 1802 are extracted. Among these, the rectangle 1802 is the same as the rectangle extracted from the first document, and is therefore excluded from the contour candidates of the second document. Accordingly, the outline candidate of the second document is only the rectangle 1801.

  In step S <b> 1704, the planar document image photographing unit 411 determines whether there is one type of document outline candidate. In the example of FIG. 18, since there is only one type of outline candidate for the second document (YES in S1704), the processing from S904 is performed. On the other hand, since there is no contour candidate for the third document (NO in S1704), the process proceeds to S1705. In step S1705, the flat document image photographing unit 411 cannot specify one type of document outline candidate, so the GUI component generation / display unit 414 displays a message notifying the user of the fact and ends the process. . FIG. 18G shows an example of a message notified to the user.

  As described above, according to the present embodiment, when only the camera image is used and the document area cannot be specified, it is possible to instruct the user to place the document again. In the present embodiment, when there is no contour candidate, a message is notified to the user and the process is terminated. However, other embodiments are possible. For example, when there are a plurality of document outline candidates or none exist, the document outline candidates up to that point may be displayed one by one, and the user may select a document outline candidate. .

<Embodiment 4>
In the third embodiment, a method is provided in which only the camera image is used and the user is instructed to reposition the original when the original area cannot be specified. However, in this embodiment, a place where the original area can be specified is calculated in advance. And provide a way to instruct the user where to place it.

  FIG. 19 is a flowchart of processing executed by the object detection unit 410 according to this embodiment. In this flowchart, the description of the same processing as that of the flowchart of FIG.

  In S1901, the object detection unit 410 calculates and displays the document placement position. For the first document, as shown in FIG. 20A, the main control unit 402 generates a rectangle 2001 so as to be horizontal with the document table 204, and the GUI component generation display unit 414 displays the generated rectangle 2001. To do. For the second document, as shown in FIG. 20B, the main control unit 402 generates a rectangle 2003 having the same center as the rectangle 2002 of the first document and having the angle inclined by α. The angle α needs to be an angle at which the contour of the document can be reliably specified. In this example, α is 30 °. The GUI component generation / display unit 414 displays the generated rectangle 2003. Similarly, for the third and subsequent originals, the main control unit 402 generates a rectangle whose angle is inclined by α, and the GUI component generation display unit 414 displays the generated rectangle.

  In step S1902, the object detection unit 410 acquires one frame of the image from the camera unit 202 via the camera image acquisition unit 407. In S1903, object detection unit 410 saves the current frame camera image as the previous frame camera image.

  As described above, according to the present embodiment, it is possible to provide a method of calculating in advance a place where a document area can be specified using only a camera image and instructing the user where to place the document area. In the present embodiment, a rectangle is displayed before the first original is placed, but other forms are also possible. For example, the user may freely place a document without displaying the first document in a rectangular form, and the position where the second and subsequent documents are placed may be indicated by a rectangular display.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (15)

A scanning system for acquiring an image of an object placed on a placement unit,
Image acquisition means for acquiring the image of the placement section and its vicinity;
Detection means for detecting placement of an object on the placement portion based on a change in images continuously obtained by the image obtaining means;
Extraction means for extracting the region of the object from the image acquired by the image acquisition means;
Conversion means for converting the region of the object extracted by the extraction means into a predetermined coordinate system;
Deriving means for deriving a contour of the region of the object converted by the converting means;
When a new object different from the object is further placed on the placement unit, the extraction unit, the conversion unit, and the derivation unit are configured so that the new object is placed on the placement unit. A scanning system characterized in that a contour after placement of the new object is derived using a correspondence relationship between feature points included in the area of the new object obtained from the images before and after being detected by the detection means. .   The scanning system according to claim 1, wherein the new object is placed on the object placed on the placement unit.   Comparing the contour size of the object area derived by the deriving means with a standard paper size, and correcting the contour of the object area to the standard paper size when the difference is less than or equal to a predetermined value The scanning system according to claim 1, further comprising means.   The scanning system according to any one of claims 1 to 3, further comprising notification means for notifying that when the deriving means cannot derive the contour of the object region. The predetermined coordinate system is an orthogonal coordinate system parallel to a plane on which an object is placed in the placement unit,
The scan system according to claim 1, wherein the conversion unit converts the region of the object by projective conversion.   6. The detection unit according to claim 1, wherein the detection unit detects that the object is placed when the position of the region of the object is not changed in the continuous images acquired by the image acquisition unit. The scanning system according to any one of the above.   The scanning system according to claim 1, further comprising an instruction unit that instructs a position where the object is placed in the placement unit.   The scanning system according to claim 7, wherein when the object is placed on the placement unit in an overlapping manner, the instruction unit switches and instructs the placement position.   The scan system according to claim 7, wherein the instruction unit performs display by projection with a projector.   The scan system according to claim 1, wherein the image acquisition unit acquires an image by photographing with a camera. The image acquisition means acquires an RGB image and a distance image for the object,
The scanning system according to any one of claims 1 to 10, wherein the detection unit performs detection based on an image change in the RGB image and the distance image.   The scan system according to claim 1, wherein the object is a flat original.   2. The apparatus according to claim 1, further comprising a receiving unit configured to analyze an image continuously acquired by the image acquiring unit to receive an operation instruction by a user gesture performed in the placement unit. 13. The scan system according to any one of items 12. A control method of a scan system for acquiring an image of an object placed on a placement unit,
An image acquisition step of acquiring the image of the placement section and the vicinity thereof;
Based on the change of the image continuously acquired in the image acquisition step, a detection step of detecting the placement of the object on the mounting portion,
An extraction step of extracting the region of the object from the image acquired in the image acquisition step;
A conversion step of converting the region of the object extracted in the extraction step into a predetermined coordinate system;
A derivation step of deriving a contour of the region of the object transformed in the transformation step,
When a new object different from the object is further placed on the placement unit, the extraction object, the conversion step, and the derivation step, when the new object is placed on the placement unit, A scan characterized by deriving a contour after placement of the new object using a correspondence relationship of feature points included in the area of the new object obtained from the respective images before and after being detected in the detection step How to control the system. Computer
An image acquisition means for acquiring an image of the mounting portion and its vicinity;
Detection means for detecting placement of an object on the placement section based on a change in images continuously obtained by the image obtaining means;
Extraction means for extracting a region of the object from the image acquired by the image acquisition means;
Conversion means for converting the region of the object extracted by the extraction means into a predetermined coordinate system;
Function as derivation means for deriving the contour of the region of the object converted by the conversion means;
When a new object different from the object is further placed on the placement unit, the extraction unit, the conversion unit, and the derivation unit are configured so that the new object is placed on the placement unit. A program for deriving a contour after placement of a new object by using a correspondence relationship of feature points included in the area of the new object obtained from respective images before and after being detected by the detection means.