A method to calibrate a camera using perpendicularity of 2D lines in the target observations

Camera calibration based on point features leads the main trends in vision-based measurement systems for both fundamental researches and potential applications. However, the calibration results tend to be affected by the precision of the feature point extraction in the camera images. As the point features are noise sensitive, line features are more appropriate to provide a stable calibration due to the noise immunity of line features. We propose a calibration method using the perpendicularity of the lines on a 2D target. The objective function of the camera internal parameters is theoretically constructed by the reverse projections of the image lines on a 2D target in the world coordinate system. We experimentally explore the performances of the perpendicularity method and compare them with the point feature methods at different distances. By the perpendicularity and the noise immunity of the lines, our work achieves a relatively higher calibration precision.

Scientific RepoRts | 6:34951 | DOI: 10.1038/srep34951 easy to be fabricated. Although the 1D target takes the simplest model in geometry, the 1D calibration method generates less information compared with the 2D calibration method in one image.
A reliable method is outlined in this paper according to the 2D calibration target. As the point features are normally adopted in the 2D calibration method, the transform matrix is calculated by the relationship between the 2D projective lines in the image and the lines in the 3D space. Then, the reprojection errors of the lines are studied in order to verify the validity of the method. The point-based calibration methods are compared with the line-based method to evaluate the accuracy and the noise immunity in the calibration process. Finally, the lines on the target are reconstructed by the projective lines and the transform matrix with the camera parameters. The objective function is built and optimized by the perpendicularity of the reconstructed lines. The perpendicularity method is compared with the original methods to evaluate the performances of the approaches.

Results
In the process of the camera calibration, it is important to provide the relationship between the coordinates of the 2D image and the ones of the 3D space. A method is proposed for the camera calibration according to the perpendicularity of 2D lines in Fig. 1. A 2D target with the checkerboard pattern is adopted as the calibration object to construct the perpendicularity of 2D lines. The world coordinate system O w -X w Y w Z w is attached on the 2D calibration board and the original point of the world coordinate system O w is located at the upper left corner of the 2D calibration board. The axes O w X w and O w Y w are defined along two vertical sides of the target. The 2D calibration board is arbitrarily placed in the view filed of the camera. O c -X c Y c Z c indicates the camera coordinate system. The original point of the camera coordinate system O c is located at the optical center of the camera. O c X c , O c Y c are respectively parallel to O p X p , O p Y p of the image coordinate system. O c Z c is the optical axis that is perpendicular to the image plane. a i and b i are two perpendicular lines generated from the checkerboard pattern on the 2D target. As the world coordinate system is attached to the 2D target, a i and b i are defined in the world coordinate system. ′ a i and ′ b i are the lines in the image coordinate system corresponding to the two perpendicular lines a i and b i in the world coordinate system. The transformation from the world coordinate system to the image coordinate system is a typical projective transformation; therefore, the line ′ a i is not usually perpendicular to the line ′ b i in the image. The vectors of the two lines ′ a i and ′ b i can be solved from the observations of the camera. According to the calibration method, the transform matrix H a is generated from the 2D lines in the world coordinate system and the 2D projective lines in the image coordinate system, firstly. Then, the initial solutions of the intrinsic parameters of the camera are contributed by the transform matrices of the observations. Finally, the optimal solutions are provided by minimizing the objective function. The following experiments are divided to two aspects, the initial solution experiments and the optimal solution experiments, in order to verify the validity of the method based on the perpendicularity of 2D lines. The point-based methods proposed by Zhang 37 and Tsai 39 are adopted as the comparative methods. The two methods are 2D plane-based calibrations. An A4 paper with the checkerboard pattern is covered on the 2D target. The size of each square is 10 mm × 10 mm. Four capture distances, 400 mm, 500 mm, 600 mm and 800 mm, are chosen to study the effect of distance in the experiments, respectively. For each distance, ten images are captured to calibrate the camera. The resolution of the images is 1024 × 768. In the process of camera calibration, 32 lines are defined by the checkerboard paper in the world coordinate system. Figure 2(a-d) show the original images observed at the distances of 400 mm, 500 mm, the world coordinate system, the image coordinate system and the camera coordinate system, respectively. H a is the homography matrix from the world coordinate system to the image coordinate system. a i and b i are two perpendicular lines in the world coordinate system. ′ a i and ′ b i are the two projective lines of a i and b i in the image coordinate system. Scientific RepoRts | 6:34951 | DOI: 10.1038/srep34951 600 mm and 800 mm in the first group of images, respectively. Similarly, Fig. 2(e-h) are the second group of images at the distances of 400 mm, 500 mm, 600 mm and 800 mm, respectively.
The coordinates of the 2D projective lines are extracted by the Hough transform [40][41][42] in the images. Hough transform finds the straight line in the parameter space that is less affected by noises. So the result of the line extraction is more stable than the result of the point extraction. Figure 3 shows the results of the Hough transform in the polar coordinate system. A sinusoidal curve corresponds to a 2D point in the Cartesian coordinate system. The blue crosses symbolize the radial coordinates ρ and the angular coordinates θ of 32 lines in the polar coordinate system. The extraction results of the lines are illustrated in Fig. 4(a-h). It is obviously that the Hough transform accurately detects the lines on the calibration target.
The line-based method is compared to the point-based methods to verify the calibration validity and noise immunity. The transform matrix H a of the line projection is experimentally obtained from the coordinates of the lines in the image coordinate system and the coordinates of the lines in the world coordinate system. Then we have 43 where a ai is the line in the world coordinate system, ′ a i a is the reprojection of the line a ai by the transform matrix H a . The transform matrices of the Zhang's method and Tsai's method are denoted by H m , H t . We have 43 where Δ a ai is the error of the line-based method, Δ a mi is the error of the Zhang's point-based method with 2D plane target, Δ a ti is the error of the Tsai's point-based method with 2D plane target, ′ a i is the image line generated by the Hough transform, ′ a i a is the reprojection of the line a ai by the transform matrix H a , ′ a i m is the fitted line derived from the transform matrix H m and the reprojections of points, ′ a i t is the fitted line derived from the transform matrix H t and the reprojections of points.
The errors of the perpendicularity method, Zhang's method and Tsai's method are shown in Fig. 5(a-h). The errors of the line-based method are less than the point-based methods. The first group of experiments corresponds to Fig. 5(a-d). The mean errors and the variances are listed in Table 1. The second group of experiments corresponds to Fig. 5(e-h). The mean errors and the variances are listed in Table 2. According to the error data above, the errors in the X direction, Y direction and the root-mean-square errors of the line-based method are all far less than the errors of the point-based methods in the two groups of experiments. The errors of the line-based method vary indistinctively with the increasing distance. However, the errors of the point-based methods show the increasing trend when the distance is on the rise. The error variances of the line-based method have been compared with the error variances of the point-based methods. It is indicated that the variation range of the errors using the line-based method is smaller than the variation range of the point-based methods. The variances of the errors adopting the line-based method fluctuate in a small range with the increase of the distance. However, the variances of the errors using the point-based methods provide a significant jump with the increasing distance. The results reveal that the line-based method is less affected by the capture distance compared to the point-based methods. According to the results and analyses above, the line-based method contributes higher accuracy in the camera calibration process.
The calibration accuracy of the three methods is further analyzed by adding different levels of Gaussian noises to the original images. The variances of the added noises are 0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02 and 0.05, respectively. The average errors are identified by the root-mean-square errors of the lines. The line-based method and Zhang's and Tsai's point-based methods are compared in Fig. 6. The values of the Gaussian noises are shown by the denary logarithms and the values of the average errors are presented by the natural logarithms for the purpose of direct observation. Figure 6(a-d) show the relationship between the average errors and the values of the noises at the distances of 400 mm, 500 mm, 600 mm and 800 mm, respectively. The errors of the three methods are all on the increase with the increasing noises. The average root-mean-square errors of the ten images using the line-based method increase from 9.06 × 10 −5 to 4.10 × 10 −2 as the noises vary from 0.0001 to 0.05 at the distance of 400 mm. Nevertheless, the average root-mean-square errors of the ten images based on the Zhang's and Tsai's methods are from 3.02 × 10 −2 to 2.62 × 10 −1 and from 5.14× 10 −2 to 3.15× 10 −1 , as the noises vary from 0.0001 to 0.05 at the distance of 400 mm. Moreover, when the distance is 500 mm, the average root-mean-square errors of the ten images grow from 8.58 × 10 −5 to 3.80 × 10 −2 . The average errors of Zhang's and Tsai's methods are from 2.81 × 10 −2 to 2.52 × 10 −1 and from 5.53× 10 −2 to 3.84× 10 −1 , respectively. At the distance of 600 mm, the root-mean-square errors of the ten images using the line-based method increase from 9.88 × 10 −5 to 4.08 × 10 −2 , respectively. The average errors of Zhang's and Tsai's methods are from 3.09 × 10 −2 to 2.61 × 10 −1 and from 5.54× 10 −2 to 4.20× 10 −1 , respectively. At the distance of 800 mm, the root-mean-square errors of the ten images using the line-based method grow from 9.13 × 10 −5 to 3.93 × 10 −2 , respectively. The root-mean-square errors of Zhang's and Tsai's methods are from 3.21 × 10 −2 to 2.57 × 10 −1 and from 6.07× 10 −2 to 3.85× 10 −1 , respectively. It is evident that the average errors of the line-based method steadily increase as the noises are on the rise. However, the average errors of the line-based method grow more slowly than the errors of the point-based methods under the nine levels of noises. It proves that the line-based method provides a better noise immunity compared with the point-based calibration methods.
The calibration results are affected by noises in the camera calibration process. Therefore, the initial solution of the intrinsic parameters should be optimized to approach the real values of the parameters. The perpendicular method is proposed to solve the optimal value of the intrinsic parameter matrix. The elements of u 0 and v 0 indicate the principal point with pixel dimensions. As the principal point should theoretically coincide with the center of the image, u 0 and v 0 are chosen to evaluate the validity of the proposed optimal method. Figure 7 presents the optimal results of the initial values of the two elements u 0 and v 0 in the intrinsic parameter matrix. The dotted lines in Fig. 7 present the coordinates of the image center in the image coordinate system. Comparative experiments are performed on the perpendicularity method and Zhang's method at the different distances.
According to Fig. 7, the initial values of u 0 and v 0 using the perpendicularity method approach to the coordinates of the center points of the images with the rising number of the images. The first few points are far away from the dotted lines. However, the optimal values of the perpendicularity optimal method are all near the dotted lines. The initial values of u 0 and v 0 based on Zhang's and Tsai's methods vary a lot with the increasing numbers of the images. Moreover, the optimal values are basically near the dotted lines as the number of the images is on the rise.
The means and the variances of the initial and optimal values of u 0 , v 0 are listed in Table 3. The calibrated intrinsic parameters of the camera at the different distances are shown in Table 4  (a-h) correspond to Fig. 2(a-h), respectively. The red data are the experiment errors of the line-based calibration method. The blue data are the Zhang's point-based calibration method. The orange data are the Tsai's point-based calibration method. The red data are located below the blue and orange data, which denote that the line-based calibration method contributes the higher accuracy.  optimal methods are also close to the coordinates of the center point. However, the descending velocity of the perpendicularity method is higher than the velocity of Zhang's and Tsai's optimal methods. Moreover, the variances of the optimal values of u 0 and v 0 based on Zhang's and Tsai's methods are larger than the variances of the perpendicularity method, except the variances of the initial and optimal values of u 0 at the distance of 400 mm, the optimal value of v 0 at the distance of 500 mm, the initial value of u 0 , the initial and the optimal values of v 0 at the distance of 600 mm, and the initial value of u 0 at the distance of 800 mm.

Discussion
In the experiment results, the line-based calibration method contributes an initial solution with higher accuracy. The perpendicularity method describes a better optimization approach. In the camera calibration process, the coordinates of the geometrical features significantly affect the accuracy of the camera calibration. In the perpendicularity method, the Hough transform is employed to extract the coordinates of the lines. As the Hough transform takes the advantage of the higher noise immunity than the point extraction method, the coordinates of lines are more accurate than the coordinates of points for the geometrical features. Furthermore, the 2D lines are stable features with respect to the variable distance. However, the 2D points that are identified in a close observation are smaller in a far observation. As the noise effects are the same on the images, the point feature tends to be recognized in the different locations for different distances. Finally, as the lines pass though the feature points, the objective function adopts the perpendicularity of the lines as the optimal object, which includes more geometrical information than the feature point superposition.

Methods
The previous method adopts points as the calibration features. Consequently, Harris corner detector 38 is often chosen to extract point coordinates in the image to calibrate the camera. In this paper, the coordinates of the lines in the image coordinate system are generated from the Hough transform 40,41 . The coordinates of a random 2D point in the Cartesian coordinate system correspond to a sinusoidal curve with two parameters, the radial coordinate ρ and the angular coordinate θ, in the polar coordinate system. Thus, a line in the Cartesian coordinate system is transferred to a series of sinusoidal curves in the polar coordinate system by the Hough transform. The polar coordinates of the crossing point of the sinusoidal curves relate to the correct coordinates of the line    in the Cartesian coordinate system. Thus, Hough transform extracts the line in the image by solving the optimal values of the radial coordinate ρ and the angular coordinate θ in the parameter space and transferring them to the Cartesian coordinate system. The calibration method includes two procedures, initial solution and optimal solution. The initial solutions of the homography matrix H a and camera parameters are solved at first by the similar way to the point-based calibration method 37 . The line transform from the 2D world coordinate system to the image coordinate system is represented as 43 h jT is the j th -row of H a , H a is a 3 × 3 transfer matrix of the camera.
As the cross product of two same vectors, ′ a i and H a a i , is a zero vector 0, a 2D projective line ′ a i in the image coordinate system and 2D line a i in the world coordinate system satisfy The singular value decomposition of the matrix G is expressed by 44 The 2D line l determined by the 2D points x 1 and x 2 is carried out as The projective line l′ determined by the image points ′ x 1 and ′ x 2 is From equations (12), (13) and (15), the projective line l′ is The right of equation (14) is transferred to where ⁎ H m is the adjoint matrix of H m . From equations (14), (16) and (17), we have    The intrinsic parameters in the matrix A are considered as the initial solutions of the camera. The image information is affected by noises, illuminations, capture distance and other factors. For this reason, an objective function is constructed to solve the optimal solutions of the camera parameters. The perpendicularity of lines is not preserved under the perspective imaging. However, the property is invariable to the reconstructed lines in the world coordinate system. The parameterized lines in the world coordinate system are reconstructed by the lines in the image coordinate system and camera parameters. Then, the objective function is established by the sum of the dot products among the perpendicular reconstructed lines. Finally, the camera parameters are achieved by minimizing the objective function. According to equations (7) and (21), the relationship between the coordinates of the lines in the world coordinate system and the coordinates of the lines in the image coordinate system can be represented as 43 i i i m T