River boundary detection and autonomous cruise for unmanned surface vehicles

Abstract

The detection of river boundaries is a crucial branch of the intelligent perception of unmanned surface vehicles (USVs), it can be used to determine the driving areas of USVs, and also to ensure driving safety by limiting the effective drivable areas of USVs in the river areas. Aiming to detect the boundaries of incompletely structured river channels, this study proposes a real-time detection method for river boundaries based on a Light Detection and Ranging (LiDAR) sensor. The point clouds that are disturbed by the water surface noise are filtered firstly, and then the spatial and geometric features are extracted separately from the point cloud detected above the water surface. To prevent the error detection and missing detection, the boundary point information is predicted and calibrated in real time by Extended Kalman Filter (EKF). A planning track generation algorithm for coastal autonomous cruise without relying on high-precision maps, and a heading and distance adaptive control method by Proportional-Integral-Derivative (PID), and different driving line generation methods for driving along the narrow river and wide river are proposed respectively. The experimental data verification of river boundary detection shows that the algorithm is accurate, real-time, and robust.

1 INTRODUCTION

In recent years, with the rapid development of unmanned systems and artificial intelligence technology in the civil field, USV has been widely used in various tasks in river waters, such as surface garbage salvage [1, 2], surface oil spill removal [3], environmental mapping [4, 5], patrol tasks [6] and environmental monitoring [7, 8]. River detection provides visual perception ability for unmanned surface vehicles, which is critical for building an auxiliary unmanned navigation system. A certain degree of autonomy helps accomplish tasks efficiently and intelligently. To realize autonomous navigation in inland rivers, the detection of river boundaries is a crucial step, which not only provides the basis for simultaneous localization and mapping (SLAM) [9, 10], and obstacle avoidance [11, 12], but also provides the necessary information basis for automatic detection [15] and subsequent path planning [13, 14].

The main task of river boundary detection is to detect the boundary line between the water surface on the river and the bank. In the existing technology, three types of sensors are usually used for boundary line detection, including vision cameras [21], radar sensors [10] and LiDAR sensors [16]. Compared with millimetre wave radar, LiDAR has higher resolution and sensitivity, higher accuracy and is not vulnerable to interference. Compared with the camera, the LiDAR can detect high-precision 3D coordinate information of objects, has a wider field of vision, a long detection distance and can detect the edge of the river with good stability.

A very similar task to river boundary detection is shoreline detection, many existing shoreline detection methods are based on image vision [21-26], and the detection results are pixel-based semantic segmentation, mainly divided into water surface or non-water surface areas. Image-based river detection algorithms are vulnerable to diversification of river scenes, illumination changes and water surface reflection interference. LiDAR is an ideal sensor. The river detection algorithm based on point cloud data [17] does not need to rely on more data sets, but needs to design a robust algorithm rule. However, it is not easy to detect the river boundary with LiDAR, because the waves and floating objects on the water surface will generate noise, and the reflection capacity of the water surface is not as good as that of the road surface. When the LiDAR signal hits the water surface, a large number of signals will be absorbed [17], the detection methods of a road cannot be directly applied to the river, which brings a great challenge for the detection of river boundaries by using LiDAR.

When USVs cruise autonomously in inland rivers, path planning needs to rely on high-precision maps, which need accurate mapping, and its acquisition cost is very high. If there is no high-precision map, we must manually preset many target cruise path points, especially in the complex curved river channel, which makes the USVs not intelligent enough. The water depth at the edge of the narrow river is low, which easily cause USV to run aground, so a certain safe distance should be kept from the shore. Therefore, it is necessary to propose a low-cost automation method, which can safely and autonomously cruise along the river bank.

As shown in Figure 1, the river scenes are diverse, the river detection algorithm should be applicable not only to conventional rivers, but also to complex curved river scenes. The main challenge of river courses is irregular and unstructured scenes, such as the presence of bridges in the centre of the river, and the edges of the river always filled with branches and obstacles. The uneven river environment leads to the particularity and diversity of river boundary detection. The special challenges of river boundary detection are as follows:

1. The positions of the obstacles on the edge of the river bank are irregular and discontinuous, the extracted boundary candidate points may be noise points. The edges on both sides of the river are asymmetrical and cannot be fitted with regular straight lines;

2. The edges of the river bank are covered with many branches, by dividing the water surface area, it is difficult to distinguish where are the drivable areas and the non-drivable areas;

3. There are many blue-green algae and green plants on the water surface, it is hard to distinguish where the ground is and where the water area is through the reflection ability of point clouds;

4. For large rivers or lakes, only one side of the water bank is clearly visible, and the other side is too far away and the features are not obvious. At this time, only one side of the water bank boundary can be detected. At this time, unmanned surface vehicles need to maintain a minimum safe distance from the shore.

FIGURE 1

Examples of river environment: (a) Obstacles on the edge of the river, (b) branches on the edge of the river, (c) bridges on the river, (d) irregular river edge (right side).

In view of the above challenges, this study proposes an effective method for river boundary detection. We obtain valid point clouds above the water surface in the river area, according to the characteristics of the river scene, a multi feature fusion method is proposed to extract candidate points of river boundary. This method is not only suitable for irregular rivers but also uneven lake edges. The effective point cloud above the water surface is extracted, and a series of feature extraction steps are used to obtain the candidate points of the river boundary. The method of river boundary points tracking is used to improve the accuracy of the detection results. Finally, according to the detection results, a low-cost autonomous cruise method for driving along the river is proposed. The main contributions of this paper are as follows.

1. We propose a multi-feature river boundary detection algorithm based on LiDAR, which can effectively detect the river boundary according to the spatial distribution and geometric features of the river point clouds;

2. We propose a method for river boundary points tracking based on EKF, which can filter the false points to optimize the detection results;

3. Our method detects the bilateral and unilateral boundaries of the narrow river and the wide river, respectively, and proposes two coastal autonomous-cruising methods: driving along the riverside or along the centreline of the river;

4. We provide a planning track generation algorithm for coastal autonomous navigation by calculating the centrelines or equidistant lines, and a PID heading and distance adaptive control method for coastal driving. The proposed method does not need to rely on high-precision maps.

2 RELATED WORKS

River boundary detection is critical to the judgment of navigable areas of the river, and it is the basis for the autonomous cruise of USVs. The following is an introduction to the research methods related to river boundary detection and autonomous cruise.

2.1 River boundary line detection

Shan et al. [17] proposed a method based on deep learning to accurately detect the navigable regions of rivers by 3D point clouds, which effectively solves the problems of discontinuity and corners of the water shoreline. However, deep learning methods are extremely dependent on data and require a large amount of computation. The research on inland river boundary detection is a new field, and we have not found any other completely consistent research.

In the existing research, the methods similar to river boundary detection are mainly about the detection of road boundary on land [32-35], which Zai et al. [32] performed by generating supervoxels, using α-Shape algorithm and energy minimization algorithm based on image segmentation extract 3D road boundary. Zhang et al. [33] proposed an automatic road segmentation method based on sliding window and use 3D LiDAR sensor to detect the curb. Wang et al. [34] directly constructed a significance map on the 3D unorganized point cloud for road boundary detection. Li et al. [35] combined lane lines with obstacle information based on 3D LiDAR point clouds to adaptively extract road boundaries. These methods are applicable to the pavement with fixed slope, but not to the water surface of the river.

Another study similar to river boundary detection is shoreline detection. There are only a few water surface studies, and most of them are based on the marine environment. Dong [24] and Lan [25] et al. proposed an image based local complexity and Hough transform to realize sea-sky-line or sea-land-line detection, but Hough transform needs to balance detection accuracy and calculation speed, in addition, it is also affected by strong edge and noise interference, resulting in clutter and flicker, which usually produces false lines. Gong et al. [26] detected the shoreline by constructing a two-stage algorithm with multiple features in the image. However, the proposed method can only detect straight lines, it does not take into account the complex water surface environment with curved shore and edge covered by obstacles. Wang et al. [21] proposed a method based on gradient significance and regional growth to detect shoreline. By calculating the direction of gradient consistency, the linear features of shoreline edge are easier to be extracted, and the false lines caused by irrelevant feature interference will be avoided. However, this method has high computational complexity and its accuracy will be significantly reduced under low light and bad weather conditions. Yao et al. [22] proposed an efficient deep-learning method for the shoreline semantic segmentation of USVs. The proposed method can achieve rapid reasoning ability and maintain relatively high accuracy in the detection of marine obstacles, but it depends on visual input and is not suitable for situating diverse illumination. Zou et al. [23] proposed an edge recognition method based on shear waves for shoreline detection in infrared images. The proposed Shearlets can be used for shoreline separation and obstacle detection, but the large amount of computation of the designed neural network decoder leads to slow operation speed, and the accuracy will be reduced when there is interference on the water surface.

2.2 Autonomous cruise

Although there are many researches and applications of autonomous cruise technology, they are mainly aimed at the road surface, while the research on water surface is rare, and few scholars have studied the automatic cruise method of USVs in inland rivers. Most of the existing autonomous cruise methods are based on global path planning [18, 19], but the methods based on global path planning rely on high-precision maps, and the cost of obtaining high-precision maps is high. Xiao et al. [20] proposed a method of autonomous cruising by unmanned surface vehicles, which can automatically clean the water surface. The proposed method relies on controlling the distance between the current ship and the river bank. However, the front river bank is uncertain, the bank edge is irregular, there may be obstacles on the edge, and the distance between the ship and the bank may change abruptly.

3 RIVER BOUNDARY DETECTION

Due to the particularity of river boundary detection, we propose a multi feature fusion method. The main contents of river boundary detection and tracking methods are shown in Figure 2. We detected the spatial and geometric characteristics of point clouds on water surface. This method is not only suitable for regular boundaries, but also for irregular boundaries. Firstly, the water surface interference is filtered, the candidate points of river boundary are extracted according to the spatial and geometric features of the point clouds, and the navigable area of water surface is obtained by fitting the river boundary line with B-spline curve. Then the results of river detection are optimized by tracking the river boundary points, for narrow rivers, the left and right boundaries are detected, while for wide rivers, only the left or right single boundaries are detected.

FIGURE 2

The structure of river boundary detection and tracking.

3.1 Water surface interference filtering

The raw data by LiDAR includes many things such as buildings, trees, vegetation, water surface and bank edges. We must remove these interference points from the background. Since most of the point clouds on the river surface will be absorbed, we fill the areas without point clouds on the river surface, and the height of point clouds in these areas is considered as a fixed minimum. When there are many blue-green algae and leaves on the water surface, the water surface has the same reflective ability as the road surface, but the horizontal plane of the water surface in the river is different from the road surface, the slope (uphill or downhill) does not need to be considered. It can be approximately considered that the plane of the water surface is a flat horizontal plane. We preprocess the point cloud data scanned by the LiDAR within 20 meters of the forward wave velocity to filter the water surface interference.

The filtering water surface interference is to filter the false noise points caused by the floating objects on the water surface, such as cyanobacteria, fallen leaves and mineral water bottles. To simplify the calculation, the point clouds on the horizontal plane are filtered to obtain the effective point clouds above the horizontal plane, Suppose the coordinates of the point cloud are$$\ {{{\bf p}}}_i( {{x}_{\mathrm{i}},{y}_{\mathrm{i}},{z}_i} )$$,$$\ 0 < i \le n$$, n is the number of point clouds. The filtered point clouds set G is

$$$$\begin{equation}G\ = \left\{ {{{{\bf p}}}_i\left( {{x}_i,{y}_i,{z}_i} \right){\mathrm{|}}i \in N \wedge \left| {{z}_i - {z}_{min}} \right| > \sigma } \right\},\end{equation}$$$$(1)

where $${z}_{min}$$ is the vertical coordinate of the lowest point of the water surface, and σ is the filtering threshold of the river horizontal plate, N is the set of natural numbers. Figure 3 shows an example of the results before and after water surface interference filtering.

FIGURE 3

The results of water surface interference filtering: (a) Image taken with camera, (b) before filtering, (c) after filtering.

3.2 River boundary point extraction

The common method of river boundary point extraction is to segment the water surface and non-water surface, filter the noise point cloud on the water surface, and then cluster the point cloud generated by obstacles. The upper layer points are filtered before the algorithm is processed. The points of H meters above the water surface will be filtered out to reduce the amount of calculation because too high points do not pose an obstacle to unmanned surface vehicles. The value of H is determined by the actual installation height of the onboard LiDAR sensor. We first convert the point cloud into a bird's-eye view. After removing the interference of water surface noise point cloud, we use the 3D spatial distribution features of point cloud data of the river for rough extraction, and further extract candidate points of boundaries through geometric features, the river boundary will be accurately determined after the two steps of feature extraction.

Figure 4a shows the river channel model. Figure 4b shows the change of elevation difference in the horizontal direction of the river channel. The horizontal scanning line is perpendicular to the river bank, and it can be found that the height of the edge of the river channel changes sharply. Therefore, boundary points can be extracted by features such as height deference and slope difference threshold. Sometimes the reflection of point clouds on the water surface is limited. Figure 4c shows the broken line diagram of point cloud density in the horizontal direction when the water surface reflection of the river is insufficient. A sudden change of density difference is one of the necessary conditions for river edge points. Therefore, boundary points of the river can be extracted from the density difference between one point field and its adjacent point field in the horizontal direction.

FIGURE 4

An analysis result of a river model: (a) An example of a river model, (b) point cloud density in the horizontal direction, (c) height difference in the horizontal direction.

3.2.1 Spatial feature extraction

The structured river of urban has a relatively stable height difference and density difference mutation at the boundary, which can be used to initially extract the water bank boundary. The height and density relationship between point clouds can be handled by meshing, which can avoid a large amount of calculations caused by processing each point cloud. Firstly, the point cloud data is projected to a 2D plane (x, y), then the grid is divided into M × M grids, the height value of the point cloud is mapped to the corresponding grid, the feature difference between each grid and the adjacent grid is calculated, and the feature candidates that meet the threshold condition within the grid or between the grids are regarded as the boundary points. In addition, for the bank with a fixed slope, the edge feature candidate points are detected by slope thresholds.

Spatial feature candidate points are determined by three observations, we mathematically define these three observations as the following:

$$$$\begin{equation} \forall gri{d}_k:\left\{ { \def\eqcellsep{&}\begin{array}{@{}*{2}{l}@{}} {if\left( {{C}_k \in {D}^*\ or\ {C}_k \in {H}^*\ or\ {C}_k \in {S}^*\ } \right)}&{{\mathrm{candidate\ points}}}\\[3pt] {{\mathrm{otherwise}}}&{{\mathrm{non}} - {\mathrm{candidate\ points}}} \end{array} } \right.\end{equation}$$$$(2)

1. Point cloud density filtering

The set D* of candidates that could meet the requirements of the distribution of the point cloud is defined as

$$$$\begin{equation}\left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {\ {D}^{\mathrm{*}} = \ \left\{ {{C}_k:{T}_{N1} \le \Delta {\rho }_k \le {T}_{N2}} \right\}}\\[6pt] {\ \Delta {\rho }_k = \left| {{\rho }_{cell,k} - {\rho }_{cell,l}} \right|\ ,\ \left| {l - k} \right| = \ 1\ }\\[6pt] {\ {\rho }_{cell,k} = \frac{{{N}_k}}{{{S}_k}}\ } \end{array} } \right.\end{equation}$$$$(3)

where $$\Delta {\rho }_k$$ is the point cloud density difference between the kth grid $${C}_k$$ and its adjacent grids, l is the index number of the adjacent grid, $${\rho }_{cell,k}$$ is the density factor of the cloud distribution, $${N}_k$$ is the number of point clouds, $${S}_k$$ is the size of the grid, $${T}_{N1}$$ and $${T}_{N2}$$ are the thresholds.

• 2.

  Height difference filtering

The set of candidate points H* that satisfies the condition of height difference is defined as

$$$$\begin{equation}\left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {\ {H}^{\mathrm{*}} = \ \left\{ {{C}_k:{T}_{z1} \le \Delta {Z}_k \le {T}_{z2},ZSt{d}_k \ge {T}_{z3}} \right\}}\\[12pt] {\ \Delta {Z}_k = \left| {\overline {{Z}_k} - \overline {{Z}_l} } \right|\ ,\ \left| {l - k} \right| = \ 1}\\[12pt] {\ ZSt{d}_k = \sqrt {\frac{1}{{{n}_{{\mathrm{cell}}}}}\sum {{({z}_{k,j} - \overline {{Z}_k} )}}^2} \ }\\[12pt] {\ \overline {{Z}_k} = \frac{1}{{{n}_{{\mathrm{cell}}}}}\ \sum {z}_{k,j}} \end{array} } \right.\end{equation}$$$$(4)

where $$\Delta {Z}_k$$ is the height difference between the current grid and the adjacent grid, it is the height difference of the candidate grids at the edge of the same river bank, l is the index number of the adjacent grid, $$ZSt{d}_k$$ is the standard deviation of the height within the grid, $${z}_{k,j}$$ is the height of each point cloud, $$\overline {{Z}_k} $$ is the mean value of height, n_cell is the total number of point clouds in the grid, j is the ID of the point cloud in the grid, $${T}_{z1}$$, $${T}_{z2}$$ and $${T}_{z3}$$ are the thresholds.

• 3.

  Slope filtering

For a river bank that has a fixed slope, the threshold of the slope between the candidate grid and the adjacent grid at the edge of the river is set to filter the noise points. We define S* as the set of candidate points that meet the requirements of the slopes, the slope is calculated according to the following formula.

$$$$\begin{equation}\left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {\ {S}^* = \ \left\{ {{C}_k:{T}_{S1} \le Slop{e}_k \le {T}_{S2}} \right\}}\\[6pt] {\ Slop{e}_k = \arctan \left( {\left| {\frac{{{Z}_{i + 1} - {Z}_i}}{{\sqrt {{{({X}_{i + 1} - {X}_i)}}^2 + {{({Y}_{i + 1} - {Y}_i)}}^2} }}} \right|} \right)\ }\\[12pt] {Slop{e}_k \in \left[ {0,\frac{{{\pi}}}{2}} \right]} \end{array} } \right.\end{equation}$$$$(5)

where $$Slop{e}_k$$ is the slope of the candidate grid point and the adjacent grid point on the edge of the river, $$({X}_i,{Y}_i,{Z}_i)$$ is the median within the grid, $${T}_{S1}$$ and $${T}_{S2}$$ are the thresholds.

If the corresponding features of C_k meet the condition of the three filters, the points in this area will be reserved as candidates of the river bank edge, and the 2D location of the points (x, y) are stored to build the candidate point set with an N × 2 matrix. If the threshold range is not satisfied, the points in the window will be ignored by setting the region as an empty set.

After gridding, the local point clouds in the grid are approximately equal, and there is no need to repeat the calculation, which can reduce the amount of calculation. In our example, the size of the gird is estimated to be 10 cm × 10 cm, and the search window size and the step size will be adjusted by the width of the water bank edge. An example of preliminary candidate points extracted from spatial features is shown in Figure 5.

FIGURE 5

Result of candidate points extracted from spatial features.

The spatial features could be used to extract the rough edge points of the river bank as the edge candidates. However, there is still a certain noise around the edge of the river, especially when the front river channel changes greatly due to the pitch angle or curve caused by the waves, the noise will increase, and it is easy to cause a great interference to the extraction of the boundary points. To improve the accuracy of the fitting result of the river boundary line, we extract boundary candidate points more accurately according to the geometric distribution features of the point cloud based on the candidate grid windows that were created in the previous steps.

3.2.2 Geometric feature extraction

The grid areas extracted in the previous steps are sorted by distance, and the geometric features are extracted to obtain more accurate river edge candidate points. The points on the boundary line are extracted and added to the candidate points, and the C_left and C_right of the point set on the left and right river boundary lines are obtained respectively. The candidate point set of river boundary extracted in the previous step contains many false points, which are mainly caused by weeds, branches and stones on both banks of the river. We use the method of point cloud classification, angle and distance filtering to extract candidate points of the boundaries. Specifically, we first project the point cloud to the XOY plane, and the direction angle and distance of the extracted candidate points satisfy the following conditions:

$$$$\begin{equation}\sqrt {{{\left( {{x}_i - {x}_j} \right)}}^2 + {{\left( {{y}_i - {y}_j} \right)}}^2} \le {d}_1\end{equation}$$$$(6)

$$$$\begin{equation}\left| {\sqrt {{{\left( {{x}_i - {x}_{{\mathrm{usv}}}} \right)}}^2 + {{\left( {{y}_i - {y}_{{\mathrm{usv}}}} \right)}}^2} - \sqrt {{{\left( {{x}_j - {x}_{{\mathrm{usv}}}} \right)}}^2 + {{\left( {{y}_j - {y}_{{\mathrm{usv}}}} \right)}}^2} } \right| \le {d}_2\end{equation}$$$$(7)

$$$$\begin{equation} \left|\textit{arct}\textit{an}\left(\frac{{y}_{i}-{y}_{\mathrm{usv}}}{{x}_{i}-{x}_{\mathrm{usv}}}\right)-\textit{arct}\textit{an}\left(\frac{{y}_{j}-{y}_{\mathrm{usv}}}{{x}_{j}-{x}_{\mathrm{usv}}}\right)\right|\le {\theta}_{1} \end{equation}$$$$(8)

where d₁, d₂, θ₁ is the distance and direction angle thresholds, $${P}_i( {{x}_i,\ {y}_i} )$$, $${P}_j( {{x}_j,\ {y}_j} )$$ and $${P}_{{\mathrm{usv}}}( {{x}_{{\mathrm{usv}}},\ {y}_{{\mathrm{usv}}}} )$$ are the coordinates of two adjacent candidate points and the USV in the XOY plane.

To select the third candidate point $${P}_k( {{x}_k,\ \ {y}_k} )$$ according to the order of angles, the three adjacent points $${P}_i$$, $${P}_j$$ and $${P}_k$$ must meet the following angle thresholds at the same time

$$$$\begin{equation}\left| {arctan\left( {\frac{{{y}_i - {y}_j}}{{{x}_i - {x}_j}}} \right) - arctan\left( {\frac{{{y}_j - {y}_k}}{{{x}_j - {x}_k}}} \right)} \right| \le {\theta }_2\end{equation}$$$$(9)

where θ₂ is the distance and direction angle threshold, The points that satisfy the above threshold conditions will be regarded as the same boundary.

Finally, a series of sets C₁, C₂, …, C_n are obtained, One or two sets C_max1 and C_max2 which have the most points will be selected, and the total number of points they contained is much greater than the other point sets. $${C}_i( {i \ne {\mathrm{max}}1\ {\mathrm{and\ }}i \ne {\mathrm{max}}2} )$$, then the left and right boundary point sets C_left and C_right are obtained according to the direction judgment. Finally, curve fitting is performed for C_left and C_right respectively to obtain the boundary lines on both sides of the river. An example of accurate candidate points extracted from geometric features is shown in Figure 6.

FIGURE 6

Result of candidate points extracted from geometric features.

3.2.3 Boundary points tracking

At the actual river edge, it is impossible to always have a very clear boundary line, and it is common to break or block the river edge. We need to guess the boundary points at the current position. When the algorithm cannot detect the river boundary, we can also give the possible positions of the points on the boundary line. At the same time, the existence of obstacles will lead to many false points and noise points in the detection results. In order to eliminate noise interference and smooth the detection of river boundary points, the boundary points at the next time are predicted and updated by river boundary tracking. River boundary point tracking uses Extended Kalman Filter to track the river boundary line, and the measurement vector of the sensor to the river boundary point is expressed as

$$$$\begin{equation}{{\bf{z}}}_t = \ h\left( {{{\bf{x}}}_t} \right) + {{\bf{v}}}_t\end{equation}$$$$(10)

where $${{\bf{z}}}_t$$ is the measured value of boundary points location, The observation function $$h( \cdot )$$ defines the observation model of the river boundary points, and $${{\bf{v}}}_t$$ is the measurement noise whose mean is zero and the noise covariance is $${{\bf{R}}}_t$$.

We complete boundary point tracking through two steps: prediction and update of river boundary points.

1. River boundary points prediction

   $$$$\begin{equation}{{\bf{x}}}_t = \ f\left( {{{\bf{x}}}_{t - 1}} \right) + {{\bf{w}}}_{t - 1}\end{equation}$$$$(11)
   $$$$\begin{equation}{{\bf{P}}}_t = {{\bf{J}}}_F\ {{\bf{P}}}_{t - 1}{\bf{J}}_F^T + {\bf{Q}}\end{equation}$$$$(12)

where $${{\bf{x}}}_t$$ is the position of the river boundary point, $${{\bf{w}}}_t$$ is the process noise with zero mean, and $$f( \cdot )$$ represents non-linear state transition function of the prediction model, $${{\bf{P}}}_t$$ is the position covariance matrix at time t,$$\ {\bf{Q}}$$ is the system noise covariance matrix, and $${{\bf{J}}}_F$$ is the Jacobian matrix of the function f.

• 2.

  River boundary points update

  $$$$\begin{equation}\ {{\bf{K}}}_t = {{\bf{P}}}_t\ {\bf{J}}_H^T{\left( {{{\bf{J}}}_H{{\bf{P}}}_t{\bf{J}}_H^T + {{\bf{R}}}_t} \right)}^{ - 1}\end{equation}$$$$(13)
  $$$$\begin{equation}\ {{\bf{\hat{x}}}}_t = {{\bf{x}}}_t\ + {{\bf{K}}}_t\left( {{{\bf{z}}}_t - {{\bf{J}}}_H{{\bf{x}}}_t} \right)\end{equation}$$$$(14)
  $$$$\begin{equation}\ {{\bf{\hat{P}}}}_t = {{\bf{P}}}_t\ - {{\bf{K}}}_t{{\bf{J}}}_H{{\bf{P}}}_t\end{equation}$$$$(15)

where $${{\bf{K}}}_t$$ is the Kalman gain at time t, $${{\bf{J}}}_H$$ is the Jacobian matrix of the measurement function $$h( \cdot )$$, $${{\bf{R}}}_t$$ is the covariance matrix of measurement noise, $${{{\bf \hat{x}}}}_t$$ is the new position after the boundary point is updated, $${{\bf{\hat{P}}}}_t$$ is the updated value of the position covariance matrix.

When updating the boundary points, the current detection result $${P}_{cur}( {{x}_{cur},{y}_{cur}} )$$ and the detection result $${P}_{pre}( {{x}_{pre},{y}_{pre}} )$$ of the previous frame must meet the threshold of distance, and it will be input to the Extended Kalman Filter to update the current value. If the distance threshold value is not satisfied, the current detection result will be discarded, and the detection result of the previous step will be retained. The threshold requirement is

$$$$\begin{equation}dist\ \left( {{P}_{cur},{P}_{pre}} \right) = \left| {{x}_{cur} - {x}_{pre}} \right|\ + \left| {{y}_{cur} - {y}_{pre}} \right| \le {d}_m\end{equation}$$$$(16)

where d_m is the distance threshold between the two points.

The tracking result is shown in Figure 7, a smoother river boundary line is obtained after Extended Kalman Filter correction.

FIGURE 7

An example of calibration points with EKF.

The detailed process of river boundary point extraction is shown in Algorithm 1, which mainly includes two steps: feature boundary point extraction and tracking. In Algorithm 1, the raw LiDAR data points $${{\Omega}}_{lidar}^{{t}_k}$$ in frame number $${t}_k\ $$are filtered by multi-feature extraction steps, the set of the left and right boundary point candidate points $$C_{left}^{{t}_k}$$ and $$C_{right}^{{t}_k}$$ are finally output. The river boundary points prediction is obtained by the previous l step values $$C_{left}^{{t}_k},C_{left}^{{t}_{k - 1}}, \ldots C_{left}^{{t}_{k - l + 1}}$$ (Line 20 of Algorithm 1) or $$C_{right}^{{t}_k},C_{right}^{{t}_{k - 1}}, \ldots C_{right}^{{t}_{k - l + 1}}$$ (Line 29 of Algorithm 1).

ALGORITHM 1. River boundary point extraction

Input: The raw LiDAR data points $${{\Omega}}_{lidar}^{{t}_k}$$ in frame number $${t}_k$$
Output: The left and right boundary points $$C_{left}^{{t}_k},\ C_{right}^{{t}_k}$$
1	Project $${{\Omega}}_{lidar}^{{t}_k}$$ onto the 2D plane, and grid the river area into M × M grids
2	$${{\Omega}}_{wsif}^{{t}_k} \Leftarrow $$ Water_Surface_Interference_Filter($${{\Omega}}_{lidar}^{{t}_k}$$)
3	$${D}^* \Leftarrow $$ Density_Filter ($${{\Omega}}_{wsif}^{{t}_k}$$)
4	$${H}^* \Leftarrow $$ Height_Difference_Filter ($${{\Omega}}_{wsif}^{{t}_k}$$)
5	$${S}^* \Leftarrow $$ Slope_Filter ($${{\Omega}}_{wsif}^{{t}_k}$$)
6	$${{\Omega}}_{space}^{{t}_k} \Leftarrow $$$${D}^*\cup{H}^*\cup{S}^*$$
7	$${C}_1,\ {C}_2,\ \ldots ,\ {C}_n\ \Leftarrow $$ Get_Geo_Feature ($${{\Omega}}_{space}^{{t}_k}$$)
8	$${C}_{max1},{C}_{max2}\ \Leftarrow $$ Select_Two_Max ($${C}_1,\ {C}_2,\ \ldots ,\ {C}_n$$)
9	if angle$$( {{P}_{usv},{C}_{max1}} ) > \pi /2$$ then
10	$$C_{left}^{{t}_k} = {C}_{max1}$$
11	else if angle$$( {{P}_{usv},{C}_{max2}} ) > \pi /2$$ then
12	$$C_{left}^{{t}_k} = {C}_{max2}$$
13	end if
14	if angle$$( {{P}_{usv},{C}_{max1}} ) < \pi /2$$ then
15	$$C_{right}^{{t}_k} = {C}_{max1}$$
16	else if angle$$( {{P}_{usv},{C}_{max2}} ) < \pi /2$$ then
17	$$C_{right}^{{t}_k} = {C}_{max2}$$
18	end if
19	// Get the left boundary points
20	$$X_{left}^{{t}_k}\ $$= EKF_Predict($$C_{left}^{{t}_k},C_{left}^{{t}_{k - 1}}, \ldots C_{left}^{{t}_{k - l + 1}}$$)
21	$$\hat{X}_{left}^{{t}_k}$$=EKF_Update($$X_{left}^{{t}_k}$$)
22	for $$p_j^{{t}_k}$$ in $$\hat{X}_{left}^{{t}_k}$$
23	if dist($$p_j^{{t}_k}$$,$$\ p_j^{{t}_{k - 1}}$$) > $${d}_{m}$$ then
24	replace($${\widehat{X}}_{\textit{left}}^{{t}_{k}}(j),{p}_{j}^{{t}_{k-1}}$$)
25	end if
26	end for
27	$$C_{left}^{{t}_k}$$=$$\hat{X}_{left}^{{t}_k}$$
28	// Get the right boundary points
29	$$X_{right}^{{t}_k}$$=EKF_Predict($$C_{right}^{{t}_k},C_{right}^{{t}_{k - 1}}, \ldots C_{right}^{{t}_{k - l + l}}$$)
30	$$\hat{X}_{right}^{{t}_k}$$=EKF_Update($$X_{right}^{{t}_k}$$)
31	for $$p_j^{{t}_k}$$ in $$\hat{X}_{right}^{{t}_k}$$
32	if dist($$p_j^{{t}_k}$$,$$\ p_j^{{t}_{k - 1}}$$) > $${d}_{m}$$ then
33	replace($${\widehat{X}}_{\textit{right}}^{{t}_{k}}(j),{p}_{j}^{{t}_{k-1}}$$)
34	end if
35	end for
36	$$C_{right}^{{t}_k}$$=$$\hat{X}_{right}^{{t}_k}$$

ALGORITHM 2. River centre line generation

Input: The left and right river boundary curve equations $$\;{x_l} = \;{f_l}( y )$$ and $$\;{x_r} = \;{f_r}( y )$$, horizontal scan step h.
Output: River centre line: $$\;{x_c} = \;{f_c}( y )$$.
1	Initialize the parameter index and iterations: i = 0, m = 1000
2	Initialize ordinate of the start point:$$\;{y_0}$$
3	while i < m do
4	Scan along the horizontal line $$\;y = {y_0} + i*h$$, Get the intersection
	points $${P_{l,i}}( {{x_{l,i}},{y_i}} )$$ and $${P_{r,i}}( {{x_{r,i}},{y_i}} )$$ with the left and right boundary lines of the river,
	where $${x_{l,i}} = {f_l}( {{y_i}} )$$, $${x_{r,i}} = {f_r}( {{y_i}} )$$.
5	Calculate the midpoint coordinate of $${P_{l,i}}$$ and $${P_{r,i}}$$, which can be used as an
	approximation points for the centreline of the river.
	$${P}_{i}\Leftarrow (\frac{{x}_{l,i}+{x}_{r,i}}{2},{y}_{i})$$
	end
6	Fitting all the midpoints by cubic curve.
	$${x_c} = {f_c}( y ) \Leftarrow {P_1} \ldots {P_i} \ldots {P_m}$$

3.3 River boundary line fitting

Based on the candidate points given in the previous step, we can detect the accurate river boundary lines in this step. For candidate points, we perform a parametric coordinate transformation, and then run a curve fitting algorithm. By comparing models like straight lines, parabolas, cubic curves, spline curves etc., a better model for a river bank line is a B-spline (see Figure 8), which allows the edges with curvature changes and conforms to the convolution curve model of the actual water bank boundary.

FIGURE 8

Comparison of fitting results of different models of river bank lines: (a) parabola, (b) cubic curve, (c) spline curve.

In order to obtain a smooth river boundary line, the fitting of the candidate boundary point set $$\overset{\scriptscriptstyle\rightharpoonup}{\mathcal{P}} \buildrel \Delta \over = \{ {{{\bf{p}}}_0,{{\bf{p}}}_1, \ldots ,{{\bf{p}}}_{{N}_p}} \}$$, and the bank curve is carried out by B-spline curve fitting through the least square fitting method to ensure smoothness and feasibility. The smoothing curve equation based on k-order B-spline is parameterized as:

$$$$\begin{equation}{{\bf{p}}}_d\left( t \right)\ = \mathop \sum \limits_{i = 0}^{{N}_c} {B}_{i,l}\left( t \right){\bf{p}}_i^*\ \end{equation}$$$$(17)

where $${{\bf{p}}}_d( t )\ = \ {[x( t ),y( t )]}^T$$, $${\bf{p}}_i^* \in \overset{\scriptscriptstyle\rightharpoonup}{{\cal{\bf P}^*}},0 \le i \le {N}_c$$ is the control points, $${N}_c$$ is the control point number. $$t \in {\Omega }_t \subset \mathbb{R}\ ,{\Omega }_t = \ [ {{t}_0,{t}_{{N}_c}} ]$$, $${B}_{i,l}( t )$$ is the ith l-order B-spline basis function determined by the Cox-de Boor recursion formulas [40]:

$$$$\begin{equation}{B}_{i,1}\left( t \right)\ = \left\{ { \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} {1,}&{t \in \Omega _t^i}\\[6pt] {0,}&{t \notin \Omega _t^i} \end{array} } \right.\ \end{equation}$$$$(18)

$$$$\begin{equation}{B}_{i,l}\left( t \right)\ = \frac{{t - {t}_i}}{{{t}_{i + l - 1} - {t}_i}}\ {B}_{i,l - 1}\left( t \right) + \frac{{{t}_{i + l} - t}}{{{t}_{i + l} - {t}_{i + 1}}}{B}_{i + 1,l - 1}\left( t \right)\end{equation}$$$$(19)

where $${{\Omega}}_t^i = \ [ {{t}_i,{t}_{i + 1}} )$$, and $${t}_0 \le {t}_1 \le \cdots \le {t}_{{N}_c}$$ are non-decreasing sequences. $$\overset{\scriptscriptstyle\rightharpoonup}{{\cal{\bf P}^{\bf{*}}}} \buildrel \Delta \over = \{ {{\bf{p}}_0^{\bf{*}},{\bf{p}}_1^{\bf{*}}, \ldots ,{\bf{p}}_{{{\bf{N}}}_{\bf{c}}}^{\bf{*}}} \}$$ is the control point set of B-spline basis function, which is obtained by fitting the candidate feature points of the river bank line using the least square method.

The matrix form of the fitting equation can be described as:

$$$$\begin{equation}{\bf{S}} = {\left[ {{{\bf{H}}}^{{\bf T}}{\bf{H}}} \right]}^{ - 1}{\bf{\ }}{{\bf{H}}}^{{\bf T}}{\bf{P}}\end{equation}$$$$(20)

$$$$\begin{equation}{\bf{H}}\ = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{3}{c}@{}} {{B}_{0,l}\left( {{t}_0} \right)}& \cdots &{{B}_{{N}_c,l}\left( {{t}_0} \right)}\\[6pt] \vdots & \ddots & \vdots \\[6pt] {{B}_{0,l}\left( {{t}_{{N}_p}} \right)}& \cdots &{{B}_{{N}_c,l}\left( {{t}_{{N}_p}} \right)} \end{array} } \right]{\bf{\ }}\end{equation}$$$$(21)

where S is the matrix composed of the set of control points, H is the configuration matrix, and P is the matrix composed of the candidate point vector $${{\bf{p}}}_k$$($$0 \le k \le {N}_w$$), which are extracted in the previous step, $${N}_p$$ is the number of candidate points, $${t}_k$$ represents the position along the curve, the calculation formula is as follows:

$$$$\begin{equation}{t}_k = \left\{ { \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} {0,}&{k = 0}\\ {\mathop \sum \limits_{i = 1}^k \vert {{\bf{p}}}_{\bf{i}} - {{\bf{p}}}_{{\bf{i}} - 1}\vert,}&{k \ge 1} \end{array} } \right.{\bf{\ }}\end{equation}$$$$(22)

4 RIVERSIDE CRUISING METHOD

The technology of USV is playing a more and more important role in patrol, surveillance, mapping and water quality monitoring, and other tasks. In inland rivers, it is necessary to cruise along the river bank, for example, the problem of using USVs to salvage surface garbage along the river bank because most of the floating garbage on the surface is concentrated on the river bank edge, and USVs are used for automatic water quality monitoring along the bank. In addition, there are also autonomous patrols along the bank by USVs to disperse the fishing crowd in the prohibited fishing areas and the swimming crowd in the prohibited swimming area. When USVs are cruising autonomously, they need to plan their driving paths based on high-precision maps, but the cost of high-precision maps is very high. Therefore, it is necessary to provide a low-cost method for USVs to cruise autonomously along the river bank.

According to the river boundary lines obtained in the previous step, we propose two ways for USVs to travel along the river, including driving in the centre of the river or driving along the edge of the river. For the narrow river, the driving area is limited and there are stones on the bank, thus, it is easy to run aground. At this time, it is the safest to drive in the centre of the river channel. For the wide river, in order to be polite to other passing ships, we set it to drive on the left side or the right side and keep the minimum safe distance from the shore. Figure 9 shows the main process of our proposed riverside cruising method.

FIGURE 9

The main process of riverside cruising method.

Limited by the detection range of the LiDAR, the point cloud is sparse when the distance exceeds 50 m. The LiDAR can detect the banks on both sides at any position in the river channel whose width is less than 50 m, therefore, we define the river less than 50 m as a narrow river in the algorithm, otherwise it is a wide river.

4.1 Driving line generation

The driving line can be detected on the land surface [30], but it is not so easy on the water surface. According to the different water scenes, we calculate the equidistant curve in wide rivers and the river central line in narrow rivers.

4.1.1 Equidistant curve

For wide rivers, the Yangtze River, lakes etc., when USVs are set to drive along the bank, the river boundary on one side is detected at this time. For example, the right boundary of the river is detected when driving on the right. To ensure safety, the USV must keep a distance d from the water bank boundary, where d > d_min, d_min is the minimum safe distance between USV and the river bank. Under different river scenarios, it is determined according to whether the river edge is regular, whether there are obstacles, whether it is easy to run aground, and other factors.

In the wide-river mode, the driving path along the river is planned as the equidistant line or offset line of the river bank, and the driving line along the bank can be regarded as the equidistant curve of the river centre line or the river boundary line. Firstly, the points on the curve are sampled and the corresponding slopes and normal vectors are calculated respectively. Then, the control points on the offset line are determined by the length of the curve segment and the node spacing.

Suppose $$x\ = \ P( y )$$ is the calculated river boundary curve equation, we convert it into the parameter equation, $$C( t )\ = \ ( {x( t ),y( t )} )$$, and

$$$$\begin{equation}\left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {x\ = \ x\ \left( t \right) = \ P\left( t \right)}\\ {y\ = \ y\ \left( t \right) = \ t} \end{array} } \right.\end{equation}$$$$(23)

The offset line equation $${C}_d( t )$$ is calculated as follows

$$$$\begin{equation}{C}_d\left( t \right)\ = \ C\left( t \right) \pm d \times \frac{{\left( { - y^{\prime}\left( t \right),x^{\prime}\left( t \right)} \right)}}{{\sqrt {x^{\prime}{{(t)}}^2 + y^{\prime}{{(t)}}^2} }}\end{equation}$$$$(24)

$${C}_d( t )$$ is the equidistant line equation that determines the driving line, $$C( t )\ = \ ( {x( t ),y( t )} )$$ represents the parameter form of the curve equation of the river bank, d is the equidistant distance, which is based on different river scenarios. The offset line is determined by the river boundary line that was extracted previously, an example is shown in Figure 10.

FIGURE 10

River side driving lines using the proposed method.

To ensure a smooth transition and safe driving, the generated driving line must be parallel to the river boundary line. We need to check the accuracy of the calculation results, and verify whether the curvature and centres of the generated driving curve are equal to the river edge curve. The curvature ρ of the resulting curve function is calculated as

$$$$\begin{equation}\rho \ = \frac{{{{\left| {x^{\prime}{{\left( t \right)}}^2 + y^{\prime}{{\left( t \right)}}^2} \right|}}^{\frac{3}{2}}}}{{\left| {x^{\prime}\left( t \right)y^{\prime\prime}\left( t \right) + x^{\prime}\left( t \right)y^{\prime\prime}\left( t \right)} \right|}}\end{equation}$$$$(25)

4.1.2 River centre line

For narrow rivers, it is recommended that the USV run along the river centreline to ensure traffic safety. The calculation of the centrelines of the narrow river model includes two cases, when the banks on both sides are approximately parallel and symmetrical, the centrelines of the river are obtained from the equidistant curve when the equidistant distance d = L/2, where L is the width of the river.

When the banks on both sides are not parallel and symmetrical, it is difficult to calculate the centrelines of river boundaries, we propose a fast approximate calculation method for calculating the centrelines, to get the intersection points by creating scanning lines along the horizontal direction, and then take the midpoint of intersection points, and make curve fitting for all midpoints. As shown in Figure 11, the fitting results are used as the centreline of the river and as the planning trajectory of USV for river side cruising. The specific steps of river centre line generation are shown in Algorithm 2.

FIGURE 11

Method for generating the river centre line.

4.2 Adaptive PID controller for tracking

The motion control system of USVs is an underactuated, unstable non-linear system, and it is difficult to establish an accurate mathematical model to achieve the desired results. There are various driving methods for USVs, including single propeller and single rudder, double propeller, multi propeller and multi rudder, and pump jet. Compared to the propulsion control method of single propeller and single rudder, double propellers have higher propulsion redundancy and faster response speed. Therefore, the method proposed in this article is based on differential double propellers driving for the motion control system of the USV.

During the driving of the USV, due to the influence of water current, waves, and wind, its heading and distance deviation from the target track will increase as it moves forward. The commonly used trajectory tracking control algorithm is proportional-integral-derivative (PID) [36, 37], but the simple heading control is difficult to control the coastal driving accuracy [37], or too many parameters are difficult to adjust [36]. Therefore, we propose a simple method of heading and distance adaptation to control the USV to travel along the planned trajectory curve along the river edge.

As shown in Figure 12, $${P}_{n - 1}$$,$$\ {P}_n$$ are the sampling target points of the river centerline cruise path, the current angle of the boat sailing is θ_cur, and the angle of the target direction is θ_obj, then the angle can be obtained as follows

$$$$\begin{equation}{\alpha }_1 = \ {\mathrm{arctan(}}\left( {{y}_n - {y}_{{\mathrm{usv}}}} \right)/\left( {{x}_n - {x}_{{\mathrm{usv}}}} \right){\mathrm{)}}\end{equation}$$$$(26)

$$$$\begin{equation}{\alpha }_2 = \ {\mathrm{arctan(}}\left( {{y}_n - {y}_{n - 1}} \right)/\left( {{x}_n - {x}_{n - 1}} \right){\mathrm{)}} - {\alpha }_1\end{equation}$$$$(27)

$$$$\begin{equation}{\theta }_{{\mathrm{obj}}} = \ {{\pi}}/2 - {\alpha }_1\end{equation}$$$$(28)

FIGURE 12

Calculation symbols for heading and distance deviation.

The current heading angle deviation $${\theta }_e$$ and lateral distance deviation $${d}_e$$ are respectively

$$$$\begin{equation}{\theta }_e = {\theta }_{{\mathrm{obj}}}\ - {\theta }_{{\mathrm{cur}}}\end{equation}$$$$(29)

$$$$\begin{equation}{d}_e = \ \left( {\sqrt {{{\left( {{y}_n - {y}_{{\mathrm{usv}}}} \right)}}^2 + {{\left( {{x}_n - {x}_{{\mathrm{usv}}}} \right)}}^2} } \right)sin{\alpha }_2\end{equation}$$$$(30)

In order to keep the USV along the driving line of the river, traveling parallel to the river, and always maintaining the course parallel to the river, the PID controller is used to control the error of USV traveling along the waypoints, the input error $$e( t )$$ includes the target heading deviation and target trajectory distance deviation, the PID controller designed for the output power $${\bf{u}}( t )$$ is

$$$$\begin{equation}{\bf{u}}\left( t \right)\ = {\left[ {{F}_l,{F}_r} \right]}^T\ = {\left[ {{F}_v + {K}_l{\xi }_F,{F}_v + {K}_r{\xi }_F} \right]}^T\ \end{equation}$$$$(31)

where $${F}_l$$ and $${F}_r$$ are the power output of the left and right propellers of the USV. $${F}_v$$ is the control input parameter for the propeller to maintain the uniform speed of the USV, $${K}_l$$,$${K}_r$$ is the conversion coefficient from the left and right propellers to the offset control input of each thruster, $${\xi }_F$$ is the control input corresponding to the yaw angle and lateral trajectory error.

According to the importance of the heading deviation and the distance deviation, adjust the yaw angle and the lateral distance error as the weight of the control input, and the adjustment method is as follows:

$$$$\begin{equation}{\xi }_F\left( t \right)\ = \ \mu {\xi }_{{\mathrm{ang}}}\left( t \right) + \left( {1 - \mu } \right){\xi }_{{\mathrm{dis}}}\left( t \right)\end{equation}$$$$(32)

$$$$\begin{equation}{\xi }_{{\mathrm{ang}}}\left( t \right)\ = K_{{\mathrm{ang}}}^p\ {\theta }_e\left( t \right) + K_{{\mathrm{ang}}}^i\mathop \smallint \limits_0^t {\theta }_e\left( t \right)dt + K_{{\mathrm{ang}}}^d{\mathrm{d}}{\theta }_e\left( t \right)/dt\end{equation}$$$$(33)

$$$$\begin{equation}{\xi }_{{\mathrm{dis}}}\left( t \right)\ = K_{{\mathrm{dis}}}^p\ {d}_e\left( t \right) + K_{{\mathrm{dis}}}^i\mathop \smallint \limits_0^t {d}_e\left( t \right)dt + K_{{\mathrm{dis}}}^d{\mathrm{d}}{d}_e\left( t \right)/dt\end{equation}$$$$(34)

$$$$\begin{equation}\mu \ = \left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {{d}_{{p}_n,{p}_{n - 1}}/{d}_{usv,{p}_n},{d}_{{p}_n,{p}_{n - 1}}/{d}_{usv,{p}_n} < 1}\\[6pt] {1,\ otherwise} \end{array} } \right.\ \end{equation}$$$$(35)

where $$K_{{\mathrm{ang}}}^p$$, $$K_{{\mathrm{ang}}}^i$$, $$K_{{\mathrm{ang}}}^d$$ and $$K_{{\mathrm{dis}}}^p$$, $$K_{{\mathrm{dis}}}^i$$, $$K_{{\mathrm{dis}}}^d$$ are the proportional gain coefficient, integral gain coefficient and differential gain coefficient of the heading angle and the lateral distance, $${\theta }_e$$ and $${d}_e$$ are the yaw angle and the lateral distance deviation between the running trajectory and the planned trajectory. The deviation is mainly caused by water flow, wind direction, and traction force etc. μ is the weight coefficient, when μ > 1, it will be set to 1 because it is close to the target and the distance error is controllable, only the heading error will be considered, when μ < 1, the smaller μ means the farther from the target, the priority is given to distance, and the distance error is weighted more heavily than heading error. The multi-objective optimization search for parameters suitable for the specified path and heading keeping is realized through automatic calibration during the USV driving process.

4.3 Obstacle avoidance

During the coastal autonomous-cruising process, the USV simultaneously detect the moving obstacles. If it encounters obstacles and there is a risk of collision, it detects the navigable areas on the left or right side of the obstacle, bypasses the obstacles through the navigable areas, and then continue to resume autonomous coastal cruising. Inspired by Tong et al. [41], we calculate the membership degree of collision risk [41] to determine whether collision avoidance measures need to be implemented. Firstly, the spatial collision risk membership function $$\gamma _i^{DCPA}$$ and the temporal collision risk membership function $$\gamma _i^{TCPA}$$ of the ith obstacle are calculated respectively according to the obstacle information detected by LiDAR. Secondly, the comprehensive collision risk coefficient $$CR{I}_i$$ of the ith obstacle is calculated by combining spatio-temporal factors:

$$$$\begin{eqnarray} CR\ {I}_i = \left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {0,\ \gamma _i^{DCPA} = 0\ or\ \gamma _i^{TCPA} = 0}\\[6pt] {\lambda \gamma _i^{DCPA} + \left( {1 - \lambda } \right)\gamma _i^{TCPA},\ \gamma _i^{DCPA} \ne 0\ and\ \gamma _i^{TCPA} \ne 0} \end{array} } \right.\nonumber\\ \end{eqnarray}$$$$(36)

where λ is the weight coefficient of spatio-temporal factors, and 0 < λ < 1. When CRI is greater than the threshold γ₀, run obstacle avoidance mode and take collision avoidance measures based on the different situation of the encounter (meeting, crossing, overtaking). The collision avoidance plan includes overtaking, following, slowing down, parking etc. When CRI is less than the threshold γ₀, the coastal autonomous cruise mode will be restored.

5 EXPERIMENTAL RESULTS AND ANALYSIS

This section introduces the experimental equipment and environment of USV, the evaluation of the river boundary detection algorithm and the results of detection. Finally, the statistical results of the river boundary detection running time of USV are presented, which proves the real-time and rationality of the proposed algorithm.

5.1 Sensors and data acquisition

As shown in Figure 13, the USV used in the experimental platform is a double-hull, double-screw and stern-type structure, and the whole ship is made of marine aluminium alloy or stainless steel. The ship is equipped with a Livox Horizon LiDAR, Hikvision high-speed industrial digital cameras, and CTI P3DU inertial navigation GPS.

FIGURE 13

Experimental platform and equipment of unmanned surface vehicle. (a) Experimental equipment of the USV, (b) the main sensors of the USV.

The Livox Horizon onboard is a non-repeatable scanning robotic LiDAR, which can provide much higher resolution than traditional LiDARs [31]. As shown in Figure 14, the effective field of view (FOV) of this sensor is 81.7° horizontally and 25.1° vertically. When the reflectivity of the target object is 80%, the farthest detection distance of the middle position of the FOV is about 260 m. The farthest detection distance of the edge position of the FOV will be shortened accordingly. In addition, the river channel has a fixed range in the field of view of the LiDAR, we extract the region of interest by filtering the redundant beams, which can effectively reduce the amount of point cloud data to improve the algorithm calculation speed.

FIGURE 14

Reflection range of the solid-state LiDAR.

In our experiment, the Livox Horizon sensor is installed on the top of the USV. The Cartesian coordinate system is shown in Figure 15. The X-axis is in front of the driving, the Z-axis is perpendicular to the earth, and the Y-axis is on the left side of the driving direction.

FIGURE 15

The cartesian coordinate system of LiDAR.

5.2 Experiment condition

In order to verify the rationality and robustness of the proposed method, we have conducted experiments in various complex river environments, including straight and curved rivers, and conducted coastal autonomous cruise experiments and feasibility studies in narrow and wide river scenarios respectively. The river detection and autonomous cruise algorithm of the autonomous driving USV runs on IPC (Industrial Control Computer) in real time. The running speed of the USV experiment was 1.63 m/s. Point cloud data processing in the algorithm is based on PCL (Point Cloud Library).

The experimental site is around the Central Park in Suzhou, China. As shown in Figure 16, an example of the experimental scene is the river located in the Central Park. The edge of the rivers are highly structured and converted with different types of river areas.

FIGURE 16

The experimental river where we collected LiDAR point clouds.

5.3 Results of river boundary detection

We use the proposed algorithm to experiment in narrow and complex rivers. Figure 17 shows the detection results of the river boundary, which is displayed by MATLAB. The blue points are the point cloud data returned by the LiDAR on the river, the red line segments are the fitting results of the river boundary line, and the green line segments are the generated driving lines.

FIGURE 17

Calculation results of river boundary lines and driving lines for driving along the river side: (a–c) the results of driving line calculated with the centreline of the river; (d–f) the results of driving line calculated with the equidistant curve.

The experimental results show that the algorithm is accurate and reasonable for straight and curved rivers, and can accurately detect the boundary between water and the bank of structured river channels with different linear shapes, especially for detecting straight lines. In the scenes for curved and irregular channels, the result of detection is not as good as a flat channel. For the wide and remote river channel, the point clouds more than 50 meters away from the USV are very sparse. As a result, the fitting result of the boundary lines of the convex river channels is not as good as the smooth river channels, and the interference of the spatial distribution of obstacles and point clouds brings noises. However, after a large number of experiments, the above noise has little impact on the final river boundary fitting algorithm, which shows that the algorithm has good robustness.

5.4 Driving line tracking

We calibrate by grouping the external parameters of Lidar and GPS, then the point cloud and the position and attitude obtained from GPS/imu are fused and calibrated to obtain the environmental reconstruction results. The points of the LiDAR are in ENU format$$\ {[ {x^{\prime},y^{\prime},z^{\prime}} ]}^T$$, and we converted them to ECEF coordinates $${[ {X,{\mathrm{Y}},{\mathrm{Z}}} ]}^T$$by using the LiDAR position$${[ {X^{\prime},Y^{\prime},Z^{\prime}} ]}^T$$. The specific conversion formula is as follows [27]:

$$$$\begin{equation} \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} X\\[12pt] Y\\[12pt] Z \end{array} } \right] = \ \left[ { \def\eqcellsep{&}\begin{array}{*{20}{c}} {{v}_1}&{{v}_2}&{{v}_3} \end{array} } \right]\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {x^{\prime}}\\[12pt] {y^{\prime}}\\[12pt] {z^{\prime}} \end{array} } \right] + \left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {X^{\prime}}\\[12pt] {Y^{\prime}}\\[12pt] {Z^{\prime}} \end{array} } \right]} \end{array} \end{equation}$$$$(37)

$$$$\begin{equation}{v}_1 = \left( { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} { - sin\lambda }\\[12pt] {\cos \lambda }\\[12pt] 0 \end{array} } \right)\ ,\ {v}_2 = \left( { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} { - sin\phi cos\lambda }\\[12pt] { - sin\phi sin\lambda }\\[12pt] {\cos \phi } \end{array} } \right)\ ,\ {v}_3 = \left( { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {cos\phi cos\lambda }\\[12pt] {cos\phi sin\lambda }\\[12pt] {\sin \phi } \end{array} } \right)\ \end{equation}$$$$(38)

$$$$\begin{equation}\varphi \ = \ arctan\left[ {\left( {Z + {{e^{\prime}}}^2{z}_0} \right)/r} \right]\end{equation}$$$$(39)

$$$$\begin{equation}\lambda \ = \ arctan2\left[ {\left( {\sqrt {\left( {{x}^2 + {y}^2} \right)} - x} \right)/y} \right]\end{equation}$$$$(40)

$$( {\varphi ,\lambda } )$$ is the latitude and longitude format, r and $${e^{\prime}}^2$$ are the intermediate variables calculated according to [28] and [29]. The calculated boundary lines and generated driving lines with GPS coordinates, and the running trajectory and attitude of coastal driving are shown in Figure 18. The experiment is divided into two scenarios: wide rivers and narrow rivers, the first scenario is driving along the river centre line on a narrow river and the second scenario is driving along the equidistant curve of the right boundary on a wide river. The USV is deviated from the planned trajectory due to wind and current, but it soon recovered to the correct trajectory after adjustment. Generally, the USV basically followed the planned driving line.

FIGURE 18

Boundary line calculation and GPS tracking of driving lines: (a,b) the calculated results with GPS coordinates; (c,d) the reconstructed lines which are overlaid on the satellite maps. (a) and (c) show the case that driving along the river centre line on a narrow river; (b) and (d) show the case that driving along the equidistant curve of the right boundary on a wide river.

5.5 Evaluation of river boundary detection algorithm

The algorithm was evaluated on a river data set marked with river boundary points, including several typical river environments: straight and curved rivers, unilateral and bilateral, and structured and unstructured river boundary detection for wide and narrow rivers. Since positioning and calibration errors always exist, the distance error threshold between the detection results and the marking results of the river bank boundary points was 15 cm. The algorithm was verified on the data set, and the points with errors less than the threshold were considered as the correct detection results. We choose three quantitative indicators (Precision, Recall and F1 score) in [38] and [39] to comprehensively evaluate the algorithm.

Table 1 shows the experimental comparison results between the method with full steps in this paper and the method with only spatial features extraction without geometric features (Geo-features) extraction and the method without EKF filtering. The test environment includes different scenarios of the linear rivers and curved rivers, unilateral and bilateral river boundary detection. It can be seen that the method is reasonable and robust, and the maximum detection rate of unilateral boundary points can reach 90.35%. The detection accuracy of spatial feature fusion of geometric features is improved by 22%, which shows that geometric features enhance the extraction of linear features. In addition, the detection accuracy of river boundaries after EKF filtering is improved by 9%. This shows that EKF can filter out noise points and further optimize the detection results.

TABLE 1. Comparison results of river boundary detection using different features with our algorithm.

		Straight river		Curved river
Quantitative indicators	Extracted features	Single boundary	Double boundary	Single boundary	Double boundary
Precision	Full features	90.35%	89.92%	84.27%	78.19%
	Without Geo-features	68.21%	62.56%	48.63%	59.44%
	Without EKF	81.43%	73.80%	72.54%	65.97%
Recall	Full features	84.23%	79.63%	77.39%	70.68%
	Without Geo-features	66.49%	57.39%	46.36%	35.94%
	Without EKF	84.67%	77.36%	78.32%	67.85%
F1	Full features	87.18%	84.46%	80.68%	74.25%
	Without Geo-features	67.34%	59.86%	47.47%	44.80%
	Without EKF	83.02%	75.54%	75.32%	66.90%

In addition, we compared the proposed algorithm with other methods (Shan et al. [17]) to evaluate the performance of river boundary detection. Comparative tests were conducted on different river scenarios, including different types of river boundary lines (straight and curved rivers) and different structural types (structured and unstructured rivers). As shown in Table 2, our algorithm can achieve good performance in the case of limited data sets. Shan's method has good detection results in a straight line, but our methods have more advantages in the detection of curved and unstructured rivers, which shows that our methods can adapt to more complex environments and has strong robustness. Because our method has considered the extraction of multiple features, and used the EKF to filter the noise points on the edge of rivers.

TABLE 2. Comparison results between our methods and other methods.

		Line type		Structural type
Quantitative indicators	Methods	Straight river	Curved river	Structured river	Unstructured river
Precision	Our methods	90.14%	81.23%	91.20%	78.56%
	Shan et al. [17]	90.03%	78.31%	89.62%	76.48%
Recall	Our methods	81.93%	74.04%	82.75%	63.91%
	Shan et al. [17]	83.54%	73.69%	81.09%	62.87%
F1	Our methods	85.84%	77.47%	86.77%	70.48%
	Shan et al. [17]	86.66%	75.93%	85.14%	69.01%

We select 850 frames from the dataset to evaluate the real-time performance of the two algorithms. Figure 19 shows the comparative statistical results of the running time of each frame in the actual river detection. Although the running time of Shan's method fluctuates slightly, its average running time exceeds 100 ms, and our method's average calculation time is 66 ms, which can meet the real-time requirements of the application of USVs in inland rivers.

FIGURE 19

Running time statistics of the proposed algorithm.

6 CONCLUSIONS

This paper presents a real-time river boundary detection and autonomous cruise method for USV, which enriches the application of USV in civil areas and improves the possibility of the popularization of unmanned autonomous systems. A multi-feature fusion method for extracting candidate points of the river boundary line is proposed. The noise point cloud caused by water clutter and floating objects is filtered, then the boundary points are roughly extracted by extracting the height difference, density and slope characteristics of the point cloud, and the boundary points are precisely extracted by using the distance and angle thresholds. In addition, the EKF is used to track the river boundary points to optimize the detection results.

Based on the river detection results, a method of autonomous driving along the river side for USV is also proposed. The river side driving method calculates the centreline of the river channel or the equidistant curve as the target planning trajectory, and controls the USV to travel along the target trajectory through the adaptive PID control algorithm. The path planning generated by the autonomous cruise method does not need to rely on high-precision maps, which reduces the operating cost of the USV autonomous cruise. The comprehensive experimental evaluation clearly shows that our proposed method has good robustness and accuracy.

However, this work is based on fixed shape rivers. In future work, we will focus on solving the problem of river boundary detection in the case of complex polygonal rivers and multiple obstacles.

AUTHOR CONTRIBUTIONS

Kai Zhang: Conceptualization; Formal analysis; Investigation; Methodology; Software; Validation; Writing—original draft. Min Hu: Funding acquisition; Supervision; Writing—review & editing. Fuji Ren: Funding acquisition; Supervision. Yanwei Bao: Writing—review & editing. Piao Shi: Writing—review & editing. Daoyang Yu: Data curation; Resources

ACKNOWLEDGEMENTS

This work has been funded by the Anhui Province Key Laboratory of Affective Computing and Advanced Intelligent Machine, School of Computer Science and Information Engineering, Hefei University of Technology and the Environmental Pollution Control Research Department, Institute of Solid State Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences. The authors thank anonymous reviewers for their constructive comments, which helped improve the article. This research was funded by the National Natural Science Foundation of China (62176084 and 62176083), and the Fundamental Research Funds for the Central Universities of China (Grant nos. PA2022GDSK0068 and PA2022GDSK0066).

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Open Research

DATA AVAILABILITY STATEMENT

The data used to support the findings of this study are included within the article.