CN112235554B

CN112235554B - Method for realizing software synchronization of projector and camera

Info

Publication number: CN112235554B
Application number: CN202011103453.9A
Authority: CN
Inventors: 肖文; 郑远林; 沈晓伟; 汪俊
Original assignee: Ninth Peoples Hospital Shanghai Jiaotong University School of Medicine
Current assignee: Ninth Peoples Hospital Shanghai Jiaotong University School of Medicine
Priority date: 2020-10-15
Filing date: 2020-10-15
Publication date: 2022-05-13
Anticipated expiration: 2040-10-15
Also published as: CN112235554A

Abstract

A method for realizing synchronization of a projector and a camera through software is characterized in that pure black images are inserted into complementary patterns and continuously projected, the frame frequency of the camera is half of the refreshing frequency of the projector, and whether the projector and the camera are synchronized can be judged through the contrast of shot images. The method has the beneficial effects that: the problem that accurate time synchronization is difficult to achieve between a projector and a camera in the existing structured light three-dimensional scanning technology can be solved, so that structured light patterns which can be acquired at high speed by using a common projector and a common camera (without a hardware triggering function) can be greatly simplified, and the cost can be reduced.

Description

Method for realizing software synchronization of projector and camera

Technical Field

The invention belongs to the field of three-dimensional scanning imaging, and particularly relates to a software synchronization method between a projector and a camera.

Background

In the field of three-dimensional imaging, one typically uses a projector and a camera to construct a structured light-based three-dimensional scanning device. The principle is that a projecting device (such as a DLP projector) is adopted to project structured light with different patterns to a measured object, meanwhile, a structured light image which is modulated by the surface of the measured object and deformed is shot, and then three-dimensional shape information of the surface of the measured object is inverted through an algorithm. Each frame projected by the projector needs to display a different texture. Due to the limitation of hardware equipment, the maximum frame rate of a common projector is 60Hz or 120Hz, and for such a scanning task with high real-time requirement, a camera needs to be aligned with the frame start time and the frame end time of the projector strictly to expose correctly, which often requires that the projector and the camera realize accurate time synchronization on hardware. It is often difficult to achieve this seemingly simple goal at low cost. For the hardware implementation method, projector-triggered camera exposure or camera-triggered projector refresh is two common hardware synchronization means. The acquisition rate can reach the maximum scanning frame rate of 60Hz or 120Hz of the projector by using special projector hardware and a camera and adding a synchronous circuit, but the problem is high hardware equipment cost. Conventional projectors and cameras have no hardware synchronization circuit.

Disclosure of Invention

Aiming at the defects of the prior art, the invention solves the problem of how to realize the time synchronization of a common projector and a camera under the condition of no synchronous triggering hardware, simplifies the three-dimensional imaging system device and can correspondingly reduce the equipment cost.

The invention innovatively provides a method for synchronizing a projector and a camera only by using a software algorithm, which solves the problem that the projector and the camera are difficult to precisely synchronize time in the existing structured light three-dimensional scanning technology, and can stabilize the frequency of the acquired pattern to be 30Hz or 60Hz, so that the structured light pattern can be acquired at high speed by using a common projector and a common camera (without a hardware trigger mechanism), thereby greatly simplifying a three-dimensional scanning device and reducing the cost.

The invention is based on the existing three-dimensional scanning device, the required hardware equipment is the same, including: projectors, cameras, projection screens, and computers. The computer is respectively connected with the projector and the camera, outputs projection image information to the projector, receives image information shot by the camera, judges whether the projector and the camera are synchronous or not according to the result and carries out necessary calibration.

The method comprises the following steps:

step 1, projecting an image at the speed of S by a projector according to the sequence of a positive film, a pure black image, a negative film and the pure black image;

step 2, the camera shoots the image projected in the step 1 according to the speed of S/2 and the exposure time of 2/S;

step 3, analyzing the image shot in the step 2 by a computer, and calculating the gray scale and the contrast of the image;

step 4, the computer judges whether the projector and the camera are synchronous according to the analysis and calculation in the step 3, if so, the whole synchronization process is finished; and if the images are not synchronous, the projector is instructed to project a frame of pure black image, and then the step 1 is returned.

Further, the computer performs analysis and judgment in the

steps

3 and 4 through software.

Further, the positive film and the negative film are two complementary patterns.

Preferably, both the positive and negative films have sharp contrast patterns.

Preferably, the positive film and the negative film are both black and white patterns.

Preferably, the positive film and the negative film are both patterns similar to a chessboard, which are composed of small blocks which are staggered in black and white.

Preferably, the value of S is 60Hz or 120 Hz.

Further, the method for judging whether the projector and the camera are synchronous comprises the following steps: selecting a gray value, counting the number of pixels with the gray value, if the number of the pixels exceeds or is lower than a certain threshold value, judging the gray value to be synchronous, otherwise, judging the gray value to be asynchronous.

Preferably, a gray value 0 is selected, the number of pixels having the gray value is counted, if the number of pixels exceeds a certain threshold, synchronization is determined, otherwise, synchronization is determined.

Preferably, a gray value 128 is selected, the number of pixels having the gray value is counted, if the number of pixels is lower than a certain threshold, synchronization is determined, otherwise, synchronization is determined.

The invention can judge whether the camera and the projector are synchronous or not based on a software method, and when the camera and the projector are asynchronous, the synchronization can be recovered by only inserting a frame of pure black image before the pure black image, so that each frame of image of the projector is just shot by the camera. Thereby completing software synchronization between the camera and the projector.

Compared with the prior art, the invention has the following beneficial effects:

1. the method is realized based on software, the original hardware is not required to be changed, and the cost is not increased;

2. the method has universality. The problem of synchronization between the camera and the projector can be solved without the aid of hardware functions, and the method is suitable for all equipment;

3. the method has the advantages of easy implementation, simple operation, no change of hardware, low cost and the like, can quickly realize judgment through a corresponding algorithm, and still keeps a higher imaging refresh rate.

Drawings

FIG. 1 is a schematic diagram of a projection shot time series of an embodiment of the present application;

FIG. 2 is a diagrammatic illustration of a projection and camera calibration flow of one embodiment of the present application;

fig. 3 is a gray level histogram of two consecutive images and a series of consecutive images captured during an ideal calibration process according to an embodiment of the present application, wherein (1) (2) (3) corresponds to a synchronous case and (4) (5) (6) corresponds to an asynchronous case;

fig. 4, an image taken during calibration and a gray-scale histogram of an embodiment of the present application, where (1) and (2) are results when in synchronization and (3) and (4) are results when out of synchronization.

Detailed Description

The preferred embodiments of the present application will be described below with reference to the accompanying drawings for clarity and understanding of the technical contents thereof. The present application may be embodied in many different forms of embodiments and the scope of the present application is not limited to only the embodiments set forth herein.

The conception, the specific structure and the technical effects of the present invention will be further described below to fully understand the objects, the features and the effects of the present invention, but the present invention is not limited thereto.

In one embodiment of the present invention, the synchronization method of projection and photographing is described as follows:

a first step, as shown in projection (1) in fig. 1, the projector projects a calibration image, i.e. patterns 1-3 in the figure, and sequentially inserts a frame of pure black image in an originally continuous calibration image sequence to form an interlaced projection sequence, so that after the pure black image is intermittently inserted in the original S-speed projection, e.g. 120fps, the speed of the calibration image sequence becomes S/2, i.e. 60 fps;

in the second step, the projected image is captured with the camera at a speed of S/2 (i.e., 60Hz) and an exposure time of 2/S (i.e., 1/60S). The sequence of images captured at this time is either correctly exposed, as in capture (1) of fig. 1, where only one calibration image appears during one exposure time, the remainder being a pure black image; or is erroneously exposed, as in shot (2) of fig. 1, when portions of two calibration images appear within one exposure time to produce image overlap;

the third step: according to the gray value or the contrast value of the calibration pattern acquired by shooting, whether a frame of pure black image needs to be additionally inserted is selected, so that the effective pattern sequence is delayed for one period, as shown in fig. 1, the condition that the projection (1) becomes like the projection (2) after a frame of pure black image is additionally inserted is realized, and at the moment, the correct result can be obtained by originally wrong exposure shooting (2).

Fig. 2 is a flow chart of the whole synchronous calibration operation process. When the calibration is started, the projector projects a positive film of the calibration pattern, a pure black pattern, a negative film of the calibration pattern, and a pure black pattern in this order at the self-refresh frequency S. While the camera takes pictures at a frequency of S/2 and an exposure time of 2/S. And the computer judges whether the two are synchronous or not according to the gray value or the contrast of the actually shot image.

If the gray value or contrast of the calculated image pair exceeds the threshold value, the projection and the shooting are synchronous, and the calibration is finished. If the gray value or the contrast of the image pair obtained by calculation is lower than a threshold value, the projection and the shooting are asynchronous, at the moment, a frame is inserted after the pure black pattern at the next moment, the pure black pattern is still formed, and the next projection is carried out; and the process is circulated until the obtained projection and the shooting are synchronized.

As shown in fig. 3, fig. 3.1 is a positive image, fig. 3.2 is a negative image, both are patterns similar to a chess board composed of small blocks staggered in black and white, but the black and the white are complementary, that is, at the same position of the patterns, the black and the white of the positive and the negative are just opposite, that is, the positive is black and the negative is white, or the positive is white and the negative is black. The projector projects images according to the sequence of the positive film, the pure black pattern, the negative film of the calibration pattern and the pure black pattern at the speed of S, the camera shoots at the speed of S/2 and the exposure time of 2/S, the number of the collected gray pixels of each frame of image is counted, and whether the frame initial phase of the camera is correct is determined according to the judgment standard of the minimum gray pixel or the maximum contrast. The method specifically comprises the following steps: if time is synchronized, the image captured by the camera is a superposition of a complete positive (or negative) image and two parts of a pure black image in one shot, and the results of fig. 3.3 can be obtained on the basis of contrast analysis. If the time is not synchronized, the image captured by the camera in one shot is a superposition of a partial positive image, a partial negative image and a complete pure black image, and the overall effect of the captured image is changed from black or white to gray as shown in fig. 3.4 or 3.5, when the result shown in fig. 3.6 can be obtained in the contrast analysis. Therefore, whether the projector and the camera are synchronous can be judged by obtaining the contrast of the image.

Fig. 4 shows one of the results produced by an embodiment of the present invention. When the calibration is started, the projector and camera are substantially synchronized and only one black and white checkerboard image is exposed during the same shot integration time, as in the case of shot 1 in fig. 1. At this time, the black and white chessboard image shot by the camera is clear, as shown in fig. 4.1, and through contrast analysis, the gray value of the image pixel can be seen to be intensively distributed around the 0 value or 255 value, as shown in fig. 4.2; when the projector and the camera are asynchronous, as in shooting 2 in fig. 1, two complementary black-and-white chessboard images are partially exposed in the same shooting integration time, and the images shot by the camera are overlapped, as in fig. 4.3, so that the contrast of the images is obviously reduced, and the gray values of the pixels of the images are distributed in a median or diffusion mode, as in fig. 4.4. In this case, the above-mentioned rectification step is required, i.e. the frame of pure black image is inserted before the pure black image, so as to recover the synchronization, as shown in the projection 2 in fig. 1.

In synchronization, the image gray values are distributed around 0 value or 255 value in a centralized way, and in asynchronization, the image gray values are distributed in a median or diffusion way. Therefore, several gray values, for example, 0, 128, and 255, may be selected, the number of pixels may be counted, and then compared with a certain set threshold, the number of pixels exceeding the threshold at

gray values

0 and 255 may be determined as synchronous, and the number of pixels exceeding the threshold at gray value 128 may be determined as asynchronous.

The foregoing detailed description of the preferred embodiments of the present application. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concept. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the concepts of the present application should be within the scope of protection defined by the claims.

Claims

1. A method for realizing software synchronization of a projector and a camera is characterized by comprising the following steps:

step 1, projecting an image at the speed of S by a projector according to the sequence of a positive film, a pure black image, a negative film and the pure black image, wherein the positive film and the negative film are two complementary patterns, and the positive film and the negative film are both patterns with sharp contrast;

step 2, shooting the image projected in the step 1 by a camera according to the speed of S/2 and the exposure time of 2/S;

step 4, the computer judges whether the projector and the camera are synchronous according to the analysis and calculation in the step 3, if so, the whole synchronization process is finished; if the images are not synchronous, the projector is instructed to project a frame of pure black image, and then the step 1 is returned;

the method for judging whether the projector and the camera are synchronous comprises the following steps: selecting a gray value, counting the number of pixels with the gray value, if the number of the pixels exceeds or is lower than a certain threshold value, judging the gray value to be synchronous, otherwise, judging the gray value to be asynchronous.

2. The method of claim 1, wherein the computer performs the analyzing, calculating and judging in the step 3 and the step 4 by software.

3. The method of claim 1, wherein the positive and negative films are black and white two-color patterns.

4. The method of claim 3, wherein the positive and negative plates are each a pattern of black and white alternating squares.

5. The method of claim 1, wherein S has a value of 60Hz or 120 Hz.

6. The method as claimed in claim 1, wherein the gray value 0 is selected, the number of pixels having the gray value is counted, and if the number of pixels exceeds a certain threshold, it is determined as synchronous, otherwise it is determined as asynchronous.

7. The method as claimed in claim 1, wherein the gray value 128 is selected, the number of pixels having the gray value is counted, if the number of pixels is lower than a threshold, it is determined as synchronous, otherwise it is determined as asynchronous.