Inferring Inertial Navigation Errors from SAR Image Distortions Using a Convolutional Neural Network

2 NAVIGATION BACKGROUND

In strapdown inertial navigation systems, measurements of specific force v^b and angular rate ω^b are used to propagate the position pⁿ, velocity vⁿ, and attitude quaternion $q_b^n$ of the aircraft, via the following differential equations:

$\left[\begin{array}{c}{\dot{p}}^n\\ {\dot{v}}^n\\ {\dot{q}}_b^n\end{array}\right]=\left[\begin{array}{c}{v}^n\\ T_b^n{v}^b+g^n\\ \frac{1}{2}{q}_b^n\otimes \left[\begin{array}{c}0\\ {\omega }^b\end{array}\right]\end{array}\right]$1

where b represents a coordinate system attached to the body and n represents a locally level frame with the x-axis aligned with the nominal vehicle velocity, the z-axis aligned in the direction of gravity, and the y-axis determined by the right-hand rule. Thus, the n frame represents an AT, CT, down (D) coordinate system.

At regular intervals, an external aiding device, such as GPS, provides information to correct the navigation states, typically in the framework of an extended Kalman filter (EKF). Between these corrections, or in the absence thereof, the navigation errors accumulate over time. For straight and level flights, which are typical in SAR data collection, error growth is modeled to the first order:

$\left[\begin{array}{c}\delta p^n\left(t\right)\\ \delta v^n\left(t\right)\\ \delta {\theta }^n\left(t\right)\end{array}\right]=\Phi (t,t_0)\left[\begin{array}{c}\delta p^n\left(t_0\right)\\ \delta v^n\left(t_0\right)\\ \delta {\theta }^n\left(t_0\right)\end{array}\right]$2

Here, Φ(t, t₀) is the state transition matrix, defined as follows:

$\Phi (t,t_0)=\left[\begin{array}{ccc}{I}_{3\times 3}& I_{3\times 3}\Delta t& (v^n\times )\frac{\Delta t^2}{2!}\\ 0_{3\times 3}& I_{3\times 3}& \left(v^n\right)\times \Delta t\\ 0_{3\times 3}& 0_{3\times 3}& I_{3\times 3}\end{array}\right]$3

The relationship between the estimated, true, and error parameters is defined by additive or multiplicative relationships:

$p^n\left(t\right)={\hat{p}}^n\left(t\right)+\delta p^n\left(t\right)$4

$v^n\left(t\right)={\hat{v}}^n\left(t\right)+\delta v^n\left(t\right)$5

$q_b^n\left(t\right)=\left[\begin{array}{c}1\\ -\frac{1}{2}\delta {\theta }^n\left(t\right)\end{array}\right]\otimes {\hat{q}}_b^n\left(t\right)$6

Thus, given an initial condition for the navigation errors, the true navigation states can be recovered by propagating the errors using Equation (2) and substituting the result into Equations (4)–(6). Therefore, the task of correcting errors in the estimated trajectory is reduced to determining the nine errors at the beginning of the time interval of interest. A more detailed discussion of the inertial navigation framework and error modeling has been provided by Lindstrom et al. (2022b) and Christensen et al. (2019).

For the simple case of constant position errors, SAR image shifts with respect to an a priori SAR map provide easily exploitable information regarding position errors. When velocity and attitude errors are included, however, the mapping from initial errors to image distortions is more complex (Christensen et al., 2019; Lindstrom et al., 2022b). Previous sensitivity studies discovered a complex and seemingly ambiguous relationship among the nine initial navigation errors and the induced SAR image distortions, as summarized in Table 1 (Christensen et al., 2019). In practical navigation systems, however, the D components of position and velocity errors are easily controlled by a barometric altimeter. The absence of vertical errors, combined with relatively small sensitivity to typical attitude errors, results in SAR image distortions dominated by position and velocity errors in the AT and CT directions. Thus, in this paper, we focus on position and velocity errors.

While image distortions are characterized by shifts and blurs in different directions (see Figure 2), certain combinations of position and velocity errors can create image distortions that appear similar to the human eye, making it difficult to determine the true source of distortion and thus the true trajectory error. We hypothesize that different error combinations create subtly different image distortions that can be learned via neural networks to predict the initial navigation errors.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 2

Demonstration of shifting and blurring distortions due to navigation errors: (top) shift distortions caused by errors in CT position for CT errors (from left to right) of –3, –1.5, 0, 1.5, and 3 m; (bottom) blur distortions caused by errors in AT velocity for AT errors (from left to right) of –0.4, –0.2, 0, 0.2, and 0.4 m/s

To calculate the extent of the errors (see Table 2), the navigation system was run with GPS updates until the state covariance matrix reached a steady state. Thus, when errors were injected into the data, the errors were calculated by comparing the steady-state covariance matrices of the GPS-available data with the covariance matrices of the GPS-denied data. The GPS-available data were used for this comparison only. These initial errors were then incorporated into the state transition matrix and propagated forward in time. Once enough time had elapsed to form an aperture, an image was formed and saved for processing in the neural network.

3 SAR DATA

We demonstrate our approach on both simulated and real SAR images. Simulated data were generated from flight and radar software developed in MATLAB. The software has the capability to generate a flight trajectory, populate the ground with radar targets, and form both truth and distorted (from initial errors) SAR images via the BPA from synthesized radar data (Duersch & Long, 2015; Lindstrom et al., 2022b). We assume that the images are well-focused unless errors have been intentionally injected.

For the real data sets, radar data were obtained from flight tests of the Space Dynamics Laboratory X-band radar system (Jensen et al., 2016). A sub-centimeter-accurate, post-processed trajectory is used as truth. To generate training and test data, the flight trajectory is first distorted with initial errors. An image is then generated using the BPA according to the selected aperture length. For the simulated data, we generated data using a 5-s aperture length (sim-5-s). For the real data, we generated data using 2-s (real-2-s), 6-s (real-6-s), 10-s (real-10-s), and 15-s (real-15-s) aperture lengths. The effects of navigation errors on images differ for different aperture lengths. As the aperture length increases, the initial errors propagate for a longer duration, potentially leading to larger image distortions. This trend is more pronounced for blurring distortions, which we use to our advantage.

For both the real and simulated data sets, the operating frequency is 9.75 GHz with a bandwidth of 500 MHz. For both data sets, the pixel spacing is 30 × 30 cm. The AT resolution is a function of the radar wavelength, slant range, and synthetic aperture length; thus, changing the aperture length will change the actual AT resolution. Consequently, the best possible AT resolution may differ from the actual pixel size. When back-projection is used to form the images, a grid of points is defined with a user-specified size, and the raw data are projected onto this grid. For convenience in processing, the grid points in our data were held constant despite changes in synthetic aperture length.

For each of the simulated and real data sets, six different scenarios were studied, with varying numbers and combinations of active navigation errors, as shown in Table 3. These scenarios were selected to determine the performance of the neural network both with and without potential error ambiguities. For example, in scenario 1, no error ambiguity is expected, as errors in AT and CT positions result in shifted images in the AT and CT directions, respectively (see Table 1). However, adding the D position error in scenario 4 introduces potential ambiguity in the CT shifts. For each of the six scenarios, a total of 192, 185, 178, 175, and 177 unique target locations were considered for the sim-5-s, real-2-s, real-6-s, real-10-s, and real-15-s data sets, respectively. For each target, we generated 100 distorted images with a size of 80 × 80 pixels, paired with the corresponding navigation errors. The data sets were then sorted by targets into training, validation, and testing sets, as shown in Table 4. We train on a set of targets and perform validation and testing on an entirely different set of targets. A network’s performance on the validation data is used to select appropriate hyperparameters. After the hyperparameters are finalized, the trained network is assessed on the test data. Thus, a network’s performance on the test data reflects its ability to generalize to new locations.

The brightness of each pixel of a SAR image represents the reflectivity of all targets within that pixel’s region. The pixel intensities are inherently right-skewed and are subject to changes in magnitude based on the geometric conditions. Thus, we preprocessed the images and errors as follows. Let X be an input image, X’ the corresponding reference image, and $y_{{\alpha }_{\beta }}$ the corresponding navigation error vector. We first log-transform each pixel in each image (including the reference images) using L = log₁₀(X), as is common practice in SAR images. We then standardize each pixel in the image to ensure standardized inputs to the neural network by subtracting the image mean and dividing by the image standard deviation: $Z=\frac{L-{\mu }_L}{{\sigma }_L}$, where μ_L and σ_L are estimated on an image-by-image basis.

Each of the navigation errors $y_{{\alpha }_{\beta }}$ is measured on a different scale. Therefore, we standardize the navigation errors as $s=\frac{y-{\mu }_y}{{\sigma }_y}$ to ensure equal consideration of all navigation errors during training. We estimate μ_y and σ_y from the entire data set. In practice, the standardization of navigation parameters would use a mean of 0 (assuming unbiased navigation errors on average) and a standard deviation estimated from an EKF (Kalman, 1960).

4 NEURAL NETWORK METHODS

The parameters (i.e., weights and biases) in a neural network are learned by minimizing a cost or loss function. For our loss function, we used the average mean squared error (MSE) across all considered standardized initial errors for training and testing:

$MSE=\frac{1}{mn}\sum _{\alpha =1}^n\sum _{\beta =1}^m{(s_{{\alpha }_{\beta }}-{\hat{s}}_{{\alpha }_{\beta }})}^2$7

where m is the number of error states considered, n is the number of training images in the data, $s_{{\alpha }_{\beta }}$ is the true standardized error of the β-th error and the α-th image, and ${\hat{s}}_{{\alpha }_{\beta }}$ is the corresponding error estimate determined by the neural network. We also used L2 regularization or weight decay to mitigate overfitting.

The loss function of a neural network is typically minimized by using some variant of the stochastic gradient descent (SGD), in which the gradient of the loss function with respect to the parameters is calculated via back-propagation. The Adam optimizer is an adaptive SGD variant that often has superior performance and is easier to tune (Kingma & Ba, 2015). We train our networks using AdamW, a variation on the Adam optimizer that improves the performance by decoupling the weight decay from the gradient-based update (Loshchilov & Hutter, 2017).

Once all of the errors are normalized, the MSE for any prediction can be compared against the value of 1 as a baseline: if the network guesses that the initial error state is 0 or randomly guesses with a mean 0 and a similar standard deviation, then the network will have an estimated MSE of 1. This comparison serves as a benchmark for learning, where an MSE of less than 1 indicates that the neural network is learning relevant information for this task.

CNNs are a special type of neural network in which some notion of spatial structure is assumed, such as in images. CNNs consist of three common types of layers (O’Shea & Nash, 2015). The first is a convolutional layer, in which neurons are arranged in three dimensions (width, height, and depth). A convolutional layer computes the output of neurons that are connected to local regions in the input by calculating the dot product between their weights and a specific region connected to the input volume. Thus, the output of a convolutional layer consists of the convolution of a filter or kernel with the input image (see Figure 3). The kernel, which consists of a matrix of weights, is used to extract features from the input image. The weights in each kernel are learned during training by choosing values that minimize the loss function. If the input image has three channels, then a kernel with size n has n × n × 3 weights. In each convolutional layer, multiple kernels are learned, allowing the network to extract multiple features.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 3

Example of a convolution operation

In this operation, the kernel slides across the input image, performing an element-wise multiplication with the input element (image patch) that overlaps with the kernel. The results are summed into a single output pixel in the current location (Dumoulin & Visin, 2016). This example shows a 3 × 3 kernel (filter) applied to the top-left corner of a 4 × 4 input image. In a CNN, the weights in each kernel are learned during training. This process is repeated by shifting the kernel by a fixed number of pixels to obtain the full output image.

Convolutional layers are typically followed by a pooling layer, which downsamples the output of a convolutional layer to reduce the number of parameters. The most common pooling layer is a max-pooling layer, which returns the maximum value inside a subregion of the image. CNNs typically consist of multiple convolutional and pooling layers and then end with one or more fully connected layers, as shown in Figure 4.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 4

Overview of a simple CNN architecture

This example illustrates the source image, followed by the most common building blocks of a CNN, such as convolutional layers, pooling layers, fully connected layers, and the output layer.

The use of the kernel via the convolution operation leads to shared weights that can learn features that may be present in any part of the image. The weights are shared, which reduces the total number of weights when compared to a fully connected neural network, in which each pixel of the image is treated as a node in the neuron. This approach leads to faster training times, allowing for improved learning.

In particular, deep CNNs with many layers tend to perform well on image recognition tasks (He et al., 2016; Zagoruyko & Komodakis, 2016), as they can learn higher levels of abstraction in the data (Nielsen, 2015). Deep networks generally have a greater capacity to learn complex functions compared with shallow networks. However, deep networks are more difficult to train because of vanishing gradient problems and other issues, requiring longer training times and very large data sets to be successful. To overcome this obstacle, many researchers use transfer learning. Transfer learning is a technique that can help to improve learning in a new task through the transfer of knowledge from another related task (Pan & Yang, 2009). The goal of using transfer learning is to fully learn the target task given the transferred knowledge instead of learning from scratch. This approach can often speed up the training and improve the performance of the model, especially when the sample size in the target task is small (Bar et al., 2015; Tajbakhsh et al., 2016).

A common approach to transfer learning with neural networks is to begin the learning process with a network that has been pretrained on a large data set. In this case, most of the architecture of the network for the target task is predefined based on the pretrained network. The layers close to the inputs and outputs are typically modified to conform to the target task. The weights in each of the predefined layers are then typically initialized using the weights learned by the pretrained network. Weights for modified or added layers are randomly initialized; then, all of the weights in the network are updated by training on data from the target task.

For our SAR navigation problem, we apply transfer learning by using existing deep CNN architectures that have been pretrained on large data sets (e.g., ImageNet (Deng et al., 2009)). Specifically, we considered multiple ResNet (He et al., 2016) and Wide ResNet (Zagoruyko & Komodakis, 2016) architectures pretrained on large image data sets. The ResNet architectures are CNNs that use a series of layers referred to as the “residual block.” This block allows networks to be much deeper while simultaneously mitigating the issue of a vanishing gradient, allowing us to obtain the capacity of deeper networks (He et al., 2016). For this reason, ResNet models have performed well on multiple image recognition tasks when used for transfer learning (Du et al., 2018; Liu et al., 2019; Ni & Binke, 2020).

We modified these ResNet networks to handle our input images, which consist of three-channel images with the distorted image, reference image, and difference image utilized as each of the channels, as shown in Figure 5. The three-channel image is fed into a randomly initialized convolutional layer followed by the ResNet architecture. We adapated the final layer of ResNet to our problem by replacing it with a fully connected layer with the same number of outputs as the error states. Figure 6 shows an example of the overall architecture using Wide ResNet 50_2.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 5

A three-channel image input

The input of our model consists of a three-channel image: the distorted image (Z_α), the reference image (Z₀), and the difference image (Z_α – Z₀). The proposed model is trained by using the stacked image (image input), which is treated as a different channel in the first convolutional layer.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 6

Full architecture for using transfer learning with Wide ResNet 50_2

The input consists of a three-channel image with the distorted image, reference image, and difference image (see Figure 5), which is fed into a randomly initialized convolutional layer followed by the considered ResNet architecture (e.g., Wide ResNet 50_2, ResNet-34). We adapted the final layer of ResNet to our problem by replacing it with a fully connected layer with the same number of outputs as error states. FC: fully connected; ReLU: rectified linear unit

In transfer learning, the weights of the pretrained network can either be fine-tuned (with the pretrained weights used for initialization), frozen, or randomly initialized. In our experiments, we found that fine-tuning all of the ResNet weights resulted in better performance than “freezing” any of the weights or randomly initializing them. This result is somewhat remarkable, considering that the data sets used for pretraining these networks contain natural images, which differ considerably from SAR images. This finding suggests that the pretrained network has learned general image features (e.g., lines and other shapes) that are also present in SAR images, which is consistent with other findings in the literature (Tajbakhsh et al., 2016; Yosinski et al., 2014).

We selected the Wide ResNet 50_2 network for our final model because it outperformed all other models (see Table 11 in Appendix B) and because its wide (and relatively shallow) nature enables faster training compared with other ResNet architectures. We considered two approaches for training the model with the real data. In the first approach, we train the model using only the training set from the data described in Table 4. In the second approach, we use transfer learning by pretraining the network (which is already pretrained on other images) on the training set of simulated data; using the resulting parameters as a starting point, we then train the model using the training set from the real data. The second approach is attempted to determine whether pretraining using simulated data can improve the performance on real data; the effects of this approach are discussed further in the next section.

5 EXPERIMENTAL RESULTS

Here, we present our experimental results obtained using the Wide ResNet 50_2 network. Results using other models are presented in Appendix A and B. We applied our model to simulated and real data, and the results were quantified in terms of the MSE (see Equation (7)) obtained on the test data for each scenario.

5.2 Real Data

Table 6 shows the results obtained for the real data sets using the proposed model. For scenarios 1 and 4, the best performance was obtained for a 2-s aperture. The network performs particularly well in scenario 1, with a low test MSE (see also Figure 7). We note that ambiguity exists in CT shifts for scenario 4, due to either CT or D position errors. The network, however, is able to identify some subtleties in the distorted SAR images and at least partially attribute the effect to the corresponding error. Figure 8 shows the distribution of the error states before and after estimation of the navigation errors for scenarios 1 and 4. The resulting error distributions after estimation are more concentrated around zero, indicating that the neural network is able to improve our knowledge of the initial error states. Furthermore, no outliers are introduced by the neural network. However, as was the case for the simulated data set, increasing the number of ambiguous error sources results in degraded overall performance, with a test MSE near unity for cases in which little information is learned by the network.

View this table:

• View inline
• View popup

TABLE 6

Model Performance for Each Error State for Scenarios 1-6 of the Real Data Sets for a 2-s, 6-s, 10-s, or 15-s Aperture Length Using the Wide ResNet 50_2 Model

The network is able to learn many of the errors for the real data sets (MSE < 1). The bold numbers represent the smallest test MSE of the considered error state for each scenario to indicate which data set achieves the best performance.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 7

Training and validation MSE as a function of training epoch for scenario 1 for the real-2-s data set

The gap between the errors is small, suggesting that little overfitting is occurring.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 8

Distribution of error states before (blue line) and after (histogram) estimation for the real-2-s data set for scenarios 1 and 4

In both cases, the error distributions are more concentrated around zero, indicating that the network has improved our ability to estimate navigation errors. No outliers have been introduced by the network.

For some of these errors, the network may also be overfitting. Overfitting typically occurs when the gap between the training error and validation/test error is large. Figure 9 shows the training and validation MSE as a function of training epoch for scenario 2 for the real-2-s data set. Here, the average training error is very low, indicating that the network is able to accurately predict both errors on the training set. However, the validation MSE is considerably higher, suggesting that overfitting is occurring. Increasing the amount of training data, especially adding more targets, will likely reduce the tendency to overfit.

• Download figure
• Open in new tab
• Download PowerPoint

FIGURE 9

Training and validation MSE as a function of training epoch for scenario 2 for the real-2-s (left) and real-10-s (right) data sets

There is a large gap between the training and validation error in the 2-s data, suggesting that the network may be overfitting. In contrast, the gap is smaller for the 10-s data, suggesting that less overfitting occurs for the real-10-s data. The large variations in losses are due to the inherent randomness of the SGD.

Most of the errors that caused issues for the network on the real-2-s data are associated with image blurring. For example, as shown in scenario 2 for the real-2-s data set, the network was not able to characterize blur distortions despite the lack of error ambiguity. Because blurring errors are more apparent with longer flight trajectories, we hypothesized that increasing the aperture length to emphasize the blurring distortion in the images would improve the performance.

Indeed, we found that as the aperture length increased from 2 s to 10 s in scenario 2, the MSE for the AT velocity error decreased by more than 25%, suggesting that the neural network successfully characterized the effect of blurs for the larger apertures. Notably, the neural network also improved its estimation of the CT velocity error by a factor of 2.

Further, in Figure 9, we see that the gap between the training and validation error for scenario 2 for the real-10-s data is smaller than the gap shown on the figure for the real-2-s data, suggesting that less overfitting occurs on the real-10-s data. In fact, Table 6 shows that the MSE for all scenarios on the real-10-s data set is less than 1, and the best performance in scenario 6, in which all errors are introduced, is achieved using a 10-s aperture. This result indicates that the neural network performs better when the blurring errors are more apparent with a larger aperture length.

To evaluate this trend further, we tested our proposed method using the real data sets with aperture lengths of 6 s and 15 s for scenarios 1–6. Table 6 demonstrates that as the aperture length increases, the network performs better in some settings. For instance, in scenarios 2 and 3, the performance improves slightly as the aperture length increases from 2 s to 6 s for the real data.

As the aperture length increases further, the network performs even better in scenarios 2 and 3. As the aperture length increased from 6 s to 15 s, the MSE of the AT position error decreased by more than 20% in scenario 3. Furthermore, the MSE of the AT velocity error decreased by more than 20% in scenarios 2 and 3, suggesting again that the neural network successfully characterized the effect of blurs for the larger apertures.

A larger aperture does not lead to uniform improvement in all scenarios. For scenarios 1 and 4, in which only positional errors were included, the best MSE performance was obtained for a 2-s aperture length. However, for all other scenarios in which velocity errors are introduced, either the 10-s or 15-s aperture length performed the best. Regardless, in all of the scenarios for the real-10-s and real-15-s data sets, the MSE is less than 1, which is not the case for the 2-s aperture data. These results confirm our hypothesis: As the synthetic aperture length of the real data increases (up to a point), the performance of our proposed method improves when velocity errors are included, which can induce blurring errors and other ambiguities.

We can compare our results here with those obtained by Lindstrom et al. (2022a). Lindstrom et al. (2022a) used an EKF combined with range–Doppler images that were compared with a reference image. Although the images created for this approach are different, Lindstrom et al. (2022a) used the same raw data employed in this work, with similar error statistics. Thus, we can compare our results to those obtained by Lindstrom et al. (2022a) and account for any differences in the statistics by normalizing the errors.

Because Lindstrom et al. (2022a) used altimeter measurements to control the vertical errors, their results are most analogous to our scenario 3, where we consider the AT and CT positions and velocities. We first consider the initial errors of the real data reported by Lindstrom et al. (2022a). The normalized AT position initial error is approximately 0.39, whereas the normalized CT position initial error is greater than 1. The normalized AT velocity initial error is approximately 1, and the normalized CT velocity initial error is approximately 0. The AT position and CT velocity errors compare favorably with our approach, although the CT position and AT velocity errors are worse. However, as time goes on, the AT errors tend to converge to zero whereas the CT errors do not achieve a steady state. This trend seems to contrast with our results, which showed CT errors that were lower than AT errors.

Overall, the comparison here does not show a clear winner between the two approaches. Combining our machine-learning approach with the EKF approach of Lindstrom et al. (2022a) may lead to overall improved results. We leave this for future work.