Visual Semantic Context and Efficient Map-Based Rotation-Invariant Estimation of Position and Heading

2.1 Definition of a Visual Semantic Context

Attaining a VSC begins with discretizing both the radial and azimuthal directions with respect to the pixel position of interest p₀ and the heading of reference ψ₀. Figure 3 delineates the decomposition of a given semantically segmented image into both the azimuthal and radial directions. For two integer parameters N_r and N_ψ, the azimuthal direction is discretized into N_ψ strips, each of which reaches out to a length of N_r pixels from p₀. Let us then denote a pixel designated by the k-th radial element of the j-th strip, which should be located at a distance of k pixels from the center, p₀, as follows:

1

The corresponding semantic of the pixel is given as s_jk. Note in Equation (1) that k ∈ {1, 2, ⋯, N_r} j ∈ {1, 2, ⋯, N_ψ} and ψ_j denotes the j-th discretized azimuthal direction, given as follows:

2

which yields ψ₁ = ψ₀. The operator ⌊·⌉ denotes rounding half up to the nearest integer such that ⌊x⌉ = ⌈⌊2x⌋ / 2⌉ to prevent designating interpixel locations. As to be described in Section 4, the proposed system seeks p₀ and ψ₀ by comparing pairs of VSCs.

We start with the semantic set , where S₀ denotes the null semantic and each S_i for i ∈ {0, 1, ⋯, N_s} denotes relevant semantics, . The semantics learnt from our earlier work and thus utilized in this study are buildings, S₁, and roads, S₂, yielding N_s = 2. Counting the number of i-th semantics laid on each strip gives the azimuthal distribution of the semantic. We collect the semantics into a row vector as follows:

3

which yields the VSC of the i-th semantic, where denotes an all-unity row vector of length n and δ_i represents the following:

4

We define the VSC, V, as an augmentation of Equation (3) in the column direction as follows:

5

Then, the VSC becomes a collection of azimuthal distributions of each semantic included within the designated range. It is straightforward to observe that V is a function of the center pixel position p₀, reference heading ψ₀, and parameters N_r, N_ψ of our choosing. Hence, we often explicitly denote the dependency as V = V(p₀, ψ₀; N_r, N_ψ) to indicate that a VSC encodes the information, given the design parameters. Conversely, p₀ = p(V) or ψ₀ = ψ(V) is also used when denoting the position or heading of a specific VSC.

2.2 Properties of Visual Semantic Contexts

One valuable property of the VSC, derived from its definition, is that shifting columns at either end to the opposite end yields an image rotation. We left-multiply V(p₀, ψ₀) by the following permutation matrix of a known parameter, n_ψ ∈ {1, 2,…, N_ψ}:

6

where I and 0 denote zero and identity matrices of given sizes, respectively. This multiplication yields and is equivalent to recalculating a VSC centered at the same pixel position as the semantically segmented image but rotated by 2πn_ψ / N_ψ radians. Therefore, the shift of columns in a VSC is hereafter considered as the rotation of a VSC, which is the core of VSC-based navigation and the algorithm proposed in this study.

Because δ_i is mutually exclusive among different values of i, the following holds:

7

Then, ν₀, for example, is a linear combination of ν_i for i ∈ {1, 2, ⋯, N_s} as follows:

8

Hence, V lacks (at least) one rank, i.e., rank(V) = (N_s + 1) -1 = N_s.

The characterization may be lost for a VSC with a large N_r because the context of every semantic in Equation (3) will be averaged out and become similar. Additionally, a too-small value will not capture enough semantics for uniqueness. Note that this parameter is related to the altitude and, thus, the relative scale of an aerial image compared with the database. A larger N_ψ value is better for a higher heading resolution, although there is a trade-off with the computation time. Choosing the N_r and N_ψ values that best describe the position and heading under a given condition, e.g., for the scale of an aerial image compared with the database, is an interesting problem. Yet, in this study, we assume that the values are fixed and known a priori. The authors’ prior work (Park et al., 2021) shows that VSCs sampled from two different locations exhibit considerably distinct patterns. The following section describes methods for capturing the difference between two VSCs while emphasizing the need for careful strategy in comparing two VSCs.

3.1 Rotation-Invariant Spatial Error Metrics

First, we consider each column of V as a sample drawn from a multivariate Gaussian distribution as , where μ_V is an empirically estimated mean given as follows:

9

and is an associated covariance matrix:

10

Then, one can compute the difference between the two VSCs by comparing the two corresponding multivariate Gaussian distributions. Here, ν_i(k) in Equations (9) and (10) denotes the k-th element of ν_i. Although numerous measures are available for comparing two multivariate Gaussian distributions, we adopt only those that can be calculated with minimal computation power and whose immediate realizations are due to the analytic nature of the Gaussian distribution. This approach is suitable for the purpose of this study, as we are comparing broad trends, not specific pixel values.

3.2 Rotation-Sensitive Error Metrics

In addition to rotation-invariant Gaussian statistics, it is necessary to design additional rotation-sensitive metrics to disambiguate a VSC from another rotated VSC. Such metrics can be utilized in characterizing the azimuthal error of a VSC or in describing the position of a VSC when alignment information is given. This study proposes a metric of the following form:

13

where d(·,·) denotes a row-wise (thus a semantic-wise) difference metric between the two VSCs whose candidates are given in the following paragraphs. Designing the error metric as Equation (13) ensures that every semantic is similarly distributed in order for the total difference between two VSCs to be small. We exploit several available metrics as candidates for d to analyze the error characteristics of VSCs.

3.3 Spatial and Directional Error Characteristics

Denoting the VSC centered in a semantically segmented aerial image as Ṽ, one can obtain the error of a specific VSC from a semantic-labeled map, i.e., D(Ṽ, V^DB), as a distribution in the spatial domain by using Equation (11) and in the azimuthal domain by using Equation (13). VWorld (MOLIT, 2014), which provides accurate and reliable labels for buildings and roads within South Korean territory, is used as the reference database for this study. The database precisely delineates concrete jungles and has been utilized in urban navigation; see, for instance, the work by Choe et al. (2018). Figure 4 shows a schematic diagram of the comparison process. One can iterate over various V^DBs to calculate error distributions. Figure 5 shows the resultant spatial error distribution of a VSC based on rotation-invariant error metrics. The heading information of the image is not required in this case. Figure 6 shows the spatial error distribution of a VSC based on rotation-sensitive error metrics. Note that the metrics introduced in Section 3.2 necessitate alignment information between the image and the database through the index j. Thus, in this example, it is assumed that the heading of the image is known. Figure 7 presents the azimuthal error distribution of a VSC. The centered graph is calculated from V^DB located at the true position by rotating it one tick at a time via Equation (6), i.e., multiplying J(1). The peripheral graphs are aided by V^DB at erroneous positions. All figures listed thus far are based on the reference position given in Figure 8, while the single grid of each three-dimensional figure covers an area of 2.5 m × 2.5 m.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 4

Diagram of VSC comparison between an aerial image and the database

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 5

Spatial error distribution of a VSC based on a rotation-invariant metric (Equation (11)) The minimum error is developed in the vicinity of the true position.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 6

Spatial error distribution of a VSC based on rotation-sensitive metrics (Equations (14) and (17))

The heading information is assumed to be given, which is not the usual case.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 7

Azimuthal error distributions of a VSC based on Equations (14) and (17) calculated at the true position, with adjacent positions mutually distanced by 10 m

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 8

Reference position and the surrounding semantic distribution used in calculating the error distributions

The latitude and longitude of the position are 36.325° and 127.383°, respectively, whereas the heading is 120°. For the background map hereafter, cyan, yellow, and blue represent buildings, roads, and null (neither buildings nor roads) semantics.

Motivated by the observation from error distributions that the VSC exhibits distinctive unimodal error characteristics in a local sense and that the azimuthal distribution shows a unique minimum when given accurate position information, the authors propose a visual map-based navigation system aided solely by the VSC. Notably, even if the semantic segmentation module trained in advance yields somewhat inexact and ambiguous boundaries among the different semantic classes, as shown in Figure 2, which should be a cause of disagreement between the location at which the minimum residual occurs and the true location in Figure 5 and Figure 6, minimizing Equation (11) and Equation (14) (or Equation (17)) can fix the position with sufficient accuracy. These findings validate our hypothesis that the overall semantic distribution, i.e., trend, especially that gathered in a polar-coordinated fashion rather than the detailed pixel locations, plays a critical role in characterizing the position and heading of an aerial image. However, the authors immediately found that naive minimization through an exhaustive search over a broad region of interest, i.e., over both spatial and heading domains, rarely yields a correct estimation with local convergence. It is also computationally expensive to iterate over a vast domain. Moreover, it is evident from Figure 6 and Figure 7 that estimating a vehicle position by minimizing Equation (13) necessitates prior knowledge of the heading (Park et al., 2021), and vice versa. Thus, our solution approach has two steps: marginalize the heading and narrow down the search pool into more probable regions through a coarse rejection scheme and then minimize the error through an exhaustive search over smaller and more likely domains. Meanwhile, the metrics in Equations (14) and (17) show a negligible difference in characterizing the heading of an image, whereas the former exhibits smoother behavior, as shown in Figure 7. Therefore, this study discards Equation (17) and selects Equation (14) as a reference rotation-sensitive error metric.

4.1 Problem Definition

The problem of visual map-based aerial navigation presented in this study is posed as to find as follows:

18

where is the region of interest, which may include the entire database or smaller areas designated by other means, such as estimated uncertainties of the INS solution, and D(·,·) denotes a symbolic representation of the difference between two input VSCs. Details will be given in Section 4.3 and Algorithm 1. The goal is to recover , the pixel position on the database corresponding to the center of the aerial image, and the heading of the aerial image that give the best match to the database. can immediately be converted to the absolute vehicle position by a trivial conversion based on the metadata of the geo-referenced map. In this study, we assume that the deduced reckoning of the INS had been propagated since the loss of the GNSS, and hence, the search region of interest is designated as a considerably vast area.

• Download figure
• Open in new tab
• Download powerpoint

ALGORITHM 1

Two-stage VSC error minimization

As mentioned earlier, an exhaustive minimization over the spatial and azimuthal domains, i.e., , often falls into a local solution. As is evident from Figure 6, this occurrence is primarily due to the multimodal characteristic of VSCs from the global perspective, which depends on the situation, e.g., the presence of repeated buildings or a certain pattern. Compared with the sharp-bordered database, uneven boundaries across the semantics obtained from the learned semantic segmentation module also contribute to the phenomenon. Figure 2 delineates this situation. Moreover, such an exhaustive approach, which is an inherent drawback of database-referenced matching, requires an excessive computational burden that grows exponentially with respect to the dimensions of the search domain. Thus, to efficiently estimate the position and heading, it is necessary to either exploit prior knowledge or exclude ambiguous states that accidentally possess a high likelihood through a rejection scheme. The former approach entails a sequential recursive formulation, i.e., Bayesian filtering, which necessitates an explicit notion of uncertainty so that the system becomes describable. However, the degree of variation in Equations (11) and (14) depends on both the pattern of Ṽ and the semantic distribution around the surrounding regions. Thus, these parameters must be chosen in a heuristic and conservative sense. In contrast, a rejection scheme is capable of fixing states in a stand-alone fashion when associated with the proper criteria. Such a scheme can be later integrated into the master navigation filter as a supportive navigation source under a loosely coupled framework. This study focuses on the latter approach in order to achieve a self-contained navigation system. However, this choice immediately inherits the chronic drawback of matching problems based on an exhaustive search, yielding heavy computations. To alleviate the issues of both local convergence and inefficient minimization, our approach proceeds with a rough minimum search over the position domain by marginalizing the heading, immediately followed by a refinement of accepted candidates.

4.2 Rejection Scheme

This study introduces spatial filtering and thus a reduction of search space using a two-sample K-S test (Massey Jr, 1951). This test is a statistical hypothesis test whose null hypothesis is that the two underlying probability distributions from which two given sets of scalar samples are drawn are identical. Considering each semantic row of a VSC, i.e., Equation (3), as a set of samples drawn from an underlying, but unknown, distribution, a specific V^DB is either accepted or rejected according to the results of multiple K-S tests with Ṽ. Once rejected, p(V^DB) is excluded from . Note that this statistical approach is based on our research hypothesis that a broad semantic trend plays an important role in characterizing aerial footage.

The null hypothesis, i.e., that two sets of samples ν⁽¹⁾ and ν⁽²⁾ are drawn from the same distribution, is rejected at level α when the supreme (maximal) difference, e(ν⁽¹⁾, ν⁽²⁾), between the two respective empirical distribution functions and satisfies the following condition:

19

where n₁, n₂ are the number of samples of ν⁽¹⁾, ν⁽²⁾ and the coefficient c(α) determined from α denotes the following:

20

Note that the empirical distribution function, which is a cumulative distribution function, is defined for the discretized radial variable of the VSC as follows:

21

where 1_A is an indicator of event A. Because Ṽ and V^DB share the same N_ψ, i.e., n₁ = n₂ = N_ψ, Equation (19) takes the following form:

22

We utilize this test to reject less likely regions before applying the collective metric in Equation (11) according to the rationale that V^DB located at the true position, p₀, should be distributed similarly to Ṽ. The test is applied to each semantic so that V^DB, of which at least one semantic is distributed similarly to Ṽ in a statistical sense, is qualified to the next step. Put differently, a V^DB is immediately rejected when Equation (19) is true for all i ∈ {0, 1, ⋯, N_s}. We apply the following recursive calculation:

23

which conveniently indicates a rejection when the initial value E(V^DB, Ṽ)₋₁ is set true. Because the test does not have to specify the distribution from which the samples are drawn, it is suited to our approach when comparing various V^DBs with Ṽ. A typical significance level of 0.05 is utilized in this study. The primary benefit of rejection is that it can substantially reduce the search space and, thus, the required computation. The computational cost of the K-S test is far less than that of N_ψ exhaustive comparisons over the heading domain, e.g., as needed for Equation (14); thus, the overall computational burden can be significantly lessened.

This approach should be appropriate because the semantic image resulting from a segmentation network often has coarse boundaries between semantics, as shown in Figure 2, which do not precisely mesh with the database. Therefore, comparing the overall trend between two VSCs while accommodating some errors is more suitable for this problem than focusing on the exact individual numerical value, because rejection based on a specific threshold applied to the individual elements of Equation (3) tends to overreact in rejecting candidates. The authors found that the rejection rate is, on average, above 80% within the area designated in Figure 8. Figure 9 highlights examples of 90% and 95% rejection. Most search spaces need not be exhaustively inspected because of the rough trend comparison, while only those that necessitate a detailed numerical comparison remain. For the first case shown in Figure 9, the remaining candidates are located along roads surrounded by repeated buildings, with three-way junctions nearby. Similarly, only four edges of a block are accepted in the second case. Note that those candidates around the true position, which must not be rejected, are correctly accepted. Therefore, fixing the position without prior knowledge is feasible. It is worth mentioning that the K-S test is also rotation-invariant.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 9

Sample results of the rejection scheme: (a) the region of interest with the true location of a vehicle, (b) rejection results based on multiple K-S tests, and (c) the same results as (b) from a top view

4.3 Proposed Algorithm

As depicted in Figure 4, VSCs throughout the entire map database, i.e., V^DBs, can be calculated and stored offline, as they are not subject to down-looking images measured along the flight. The procedure is conducted only once for the operation of the proposed system. Immediately after the aerial image is fed, it is assumed that one readily obtains a semantically segmented version, i.e., Ṽ, with the help of a convolution neural network trained as described by Hong et al. (2021). Assuming the use of an industrial-grade IMU, whose position drift grows a few tens of meters in less than a minute, the INS-indicated search region, i.e., , is considered to have significant uncertainty, as represented in Table 1.

Thus, the proposed algorithm proceeds with a coarse rejection based on the scheme described in Section 4.2. The primary intention is to effectively reduce the search space and, thus, the computation time and any obvious false matching of Equation (18) through multiple K-S tests in an abstract fashion. Next, we apply the rotation-invariant comparison metric in Equation (11) to pairs of (Ṽ, V^DB) over the accepted region within the database to sort out the most probable candidates. The computation time of this step can be further reduced by choosing only N least-different V^DBs. Finally, an exhaustive minimization over the chosen VSCs is conducted by using a rotation-sensitive metric, such as Equation (14), whose naive application to the entire without the prescribed procedures is impractical and inaccurate. The detailed steps of the proposed algorithm are highlighted in Algorithm 1.

5.1 Simulation Setup

Given Figure 4, we consider images cropped from Google Maps as down-looking aerial images in this study. Although aerial images taken from a series of field (flight) tests¹ can help verify the effectiveness and robustness of the proposed method, a numerical simulation is conducted in this study using synthetic data from public maps for the purpose of feasibility testing. The requirement that aerial images must be taken from a sufficiently high altitude, in order to capture various spatial distributions of semantic classes within the image, is difficult to meet. Evaluation of the proposed system showed that this approach is suitable for its given purpose as long as the primary assumptions listed in Section 1 hold, i.e., the existence of a temporal gap and a disparity of shooting conditions between the visual map database and aerial images.

To support the arguments of temporal, particularly seasonal, invariance of the proposed approach, a semantic database constructed from past data is utilized as the reference database in this study. Meanwhile, aerial images cropped from Google Maps at the latest time available are considered as aerial images taken from the vehicle. The two asynchronous sources have a two-year temporal gap, as well as a difference in shooting conditions, such as perspective (orthogonality) and light conditions. The semantic-labeled database utilized in the experiment, as highlighted in Figure 1 and Figure 10, delineates the Daejeon district, South Korea, where the left-upper corner position is 36.326° latitude and 127.378° longitude and the right-lower corner position is 36.325° latitude and 127.400° longitude. The database is cut in half and presented in two rows in the latter figure.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 10

Test sites used in evaluating the proposed algorithm

The difficulty class to which each position belongs is highlighted in different colored shapes, while each site is enumerated from left to right.

The aerial image is immediately followed by semantic segmentation based on the module trained in the work of Hong et al. (2020) so that Ṽ is rendered; see Figure 4. One can calculate VSCs for each point in the database beforehand so that the proposed two-stage VSC minimization described in Algorithm 1 proceeds as is. For reproducibility, the parameters used in the numerical experiments and other details of the simulation are listed in Table 1.

5.2 Localization Accuracy

This section presents numerical experiments and their results to demonstrate the performance of the proposed method. The performance is analyzed under three distinctive conditions: easy, medium, and hard. Each respective field is classified in terms of the localization difficulty based on the non-repetitiveness of the surrounding semantics and the existence of azimuthally distinguishable semantic patterns, i.e., dominant semantics within a limited direction of view. The readers may refer to Figure 11 in advance for the features of each group, especially in terms of auto-correlation. The test sites, denoted as a combination of difficulty and an index, e.g., M1, are shown in Figure 10. The authors found it difficult to identify the estimation tendency and to analyze the performance when the results are enumerated over random trajectories or the entire database. Therefore, the results are presented in an organized and selected fashion to provide an understandable and consistent rationale.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 11

Multiple VSCs for various test cases: (a) semantic distribution of sampled VSCs, (b) semantic-wise auto-correlation function of each VSC

Note that high peaks of auto-correlation represent rotational symmetry with respect to specific rotation angles.

The easy group cases possess more than one non-repeating semantic pattern. Cases such as asymmetric junctions (E3, E5, E6), locations at which specific and distinctive directions of view are dominantly occupied by a single semantic class (E2, E4), and locations for which there are no similar patterns nearby (E1) are included. In short, asymmetry and uniqueness are essential characteristics of the easy class. The distribution of semantic segments for the medium class is similar to that of the easy class, while the asymmetry condition is relaxed to a weak symmetry. Either rotational or reflective symmetry exists in the medium class; thus, V^DBs whose reference headings are ψ₀ + qπ or bπ – ψ₀ for q ∈ {0.5,1,1.5,2} and b ∈ {1,2} have potential to be minimally dissimilar VSCs according to Equation (13), yielding false matches. Equiangular four-way junctions (M1, M2, M6), locations at which symmetry is formed along the road (M3, M4), and locations for which local patterns such as small T-junctions are repeated nearby (M5) are included in this class. The cases designated as hard are both ambiguous and symmetric. Examples of this group include locations inside an apartment complex in an urban area where all buildings are similarly constructed and distributed in a mutually equidistant fashion (H2, H4, H5, H6) or regions in which the symmetry is hardly distinguishable (H1, H3). This group also includes an ill-posed case that is uniformly and fully occupied by a single or two semantic classes. A typical example of such an ill-posed case is the center of a wide road or a vast null area (neither a road nor a building) (H7). The hard VSC cases are highlighted in Figure 11(a) in comparison with the easy and medium cases. To denote the symmetry or repeating characteristic of the medium and hard cases, the result of the auto-correlation function applied to each semantic element of the VSC, V^DB, is given in Figure 11(b). Note that the auto-correlation of H1 and H2 has peaks close to 1 at 90° and 180° rotation, implying rotational symmetry between the original VSC and its rotated copies. One can see from Figure 10 that these results are due to repeated buildings or equiangular junctions. Similarly, the M3 case has one peak at 180° rotation, while an easy case (E5) has no outstanding peaks, indicating rotational uniqueness.

Table 2 summarizes the norms of two-dimensional position errors of the estimate, , converted into meters, and the heading error, , in degrees for each designated case and reference position. The rejection rate of Equation (23) (for i = N_s) is also presented as a percentage with respect to the size of to denote the computation efficiency. Note that this result does not reflect the overall reduction of the computation budget, but represents an omission rate of unnecessary calculations of Equation (13) over the heading domain. Supported by rotation-invariant metrics and the rejection scheme, the proposed VSC matching algorithm is independent of the orientation of Ṽ. Nevertheless, each case is tested under two different reference headings, ψ₀. The “North” values in the first column correspond to the case of ψ₀ = 0, and the “East” values correspond to the other case, ψ₀ = π/2. The heading angles of Ṽ for each test case are randomly sampled around the true heading, i.e., uniform within [ψ₀ – π/N_ψ, ψ₀ + π/N_ψ). Therefore, the heading estimation errors of the two cases case differ by up to π/N_ψ, which should be 1° in this numerical study.

The first value in each cell (before the slash) denotes the estimation error of the proposed method, based on the procedure illustrated in Figure 4. The second value of each cell (after the slash) denotes the result obtained by the same algorithm when the semantic segmentation module is replaced with the accurate semantic label so that the proposed algorithm is tested using V^DB instead of Ṽ. These results are provided to support the idea that the accuracy degradation (such as that observed for H1, H2, or H4) is not due to the proposed algorithm but to the moderate performance of the up-to-date segmentation module, which still yields rough boundaries between semantic segments; unless the problem is ill-posed for the first location (H7), in which case the solution is unavailable. Perfect matches are achievable as long as the semantic segmentation module yields accurate results.

The localization accuracy of the proposed algorithm for the easy group is less than 2 m for the position and 2° for the heading. For the medium group, the overall estimation error is slightly larger, reaching 5 m and 6°. These estimates should be relatively accurate, considering the absence of the GNSS and the vast search domain, . In the hard cases, the position estimate accuracy is 9.8 m on average, whereas the accuracy of the extreme case rises to 26 m. The heading estimation accuracy is similar to that of the medium cases. Considering a further application of the position and heading fixing module to the integrated navigation filter, all cases result in admissible estimation errors. However, when the VSC parameter N_r is decreased to 45, a false match occurs in the H1 case, with a heading error of 173°. This result is due to the inherent ambiguity of matching approaches, from which the proposed approach is not immune. Nevertheless, the user can enhance the accuracy by adjusting system parameters, especially when the region of interest becomes more ambiguous or symmetric. From this observation, one can expect that there may be an optimal choice of VSC parameters such as N_r, N_ψ or α in terms of localization accuracy and robustness. For instance, N_r should be increased when the local pattern repeats over an area or when the VSC has multiple auto-correlation peaks so that the enclosed area is sufficiently distinct and unique.

The rejection rate is higher than 87% for all cases; thus, a significant amount of unnecessary computation is avoided, except for the H3 case, where all V^DBs along the thick road should be similar. Again, as calculating Equation (13) for every N_ψ rotation has become unnecessary for the less probable regions based on Equation (23), the computational burden is greatly reduced. Applying different α values to the algorithm will result in the acceptance of more regions and thus a higher probability of true matches; however, one must consider the trade-off with the required time.

Whereas the rotational symmetry of a specific VSC can be identified from auto-correlation results, the rejection rate itself has little to do with spatial repetitiveness or position accuracy. Instead, the distribution of the p-value of multiple K-S tests over a spatial domain or, equivalently, the difference between each side of Equation (22) should help identify positional ambiguity or spatial repetitiveness of a given VSC pattern. Consequently, a well-defined index combining the results of the auto-correlation function and the p-value distribution of the K-S test within could play a critical role in predicting the VSC matching performance, without the need for manual classification. Furthermore, such an index could indicate whether the localization accuracy will be sufficient even before the algorithm is applied. Namely, assigning a difficulty category to each position on the map database based on the number of auto-correlation peaks could help avoid false matches.

Moreover, integration with the INS can further prevent the occurrence of completely false matches via temporal inference. Here, temporal inference denotes a trust region constrained by a vehicle’s dynamic/kinematic limit (or the INS model) so that the higher-level integrated navigation system does not adopt a false match beyond this boundary. For instance, let us consider a vehicle flying at 100 m/s with an update rate of 1 Hz. In this case, a match should be false if two consecutive estimated positions differ far more than 100 m. If we operate a windowed queue of estimation results whose size is larger than 2, the exact false match case can be identified and isolated. Furthermore, the quantized estimation of the DBRN matching approach can be interpolated to a sub-pixel resolution when blended with the INS.

5.3 Scale Sensitivity

In many cases, the scale of an aerial image does not match that of the label map. In other words, the altitude at which a vehicle is flying may differ from the reference altitude at which the label database was rendered. For such situations, modifying Equation (1) by introducing a scale parameter, , yields the following:

24

We can adjust the value of γ accordingly when the scale of the ground map differs from the given footage, taking the reference altitude and resolution of both sources into account as follows:

25

where r, h denote the resolution and reference altitude, respectively, while the subscripts AI and DB indicate the semantically segmented aerial image and the semantic-labeled database, respectively. Fixing γ_DB as 1, for instance, and using the relative value of Equation (25) for γ_AI will resolve the scale difference. However, this approach cannot handle a case in which the vehicle’s altitude is inaccurate. It is well known that a barometric altimeter (BA) can accurately measure the altitude of an aerial vehicle by relating both the temperature, T, and pressure, P, to the altitude with the help of the universal gas constant, R, lapse rate, β, and gravity constant, g, as follows:

26

However, the inherent principle error of the BA (Bao et al., 2017) is represented as a combination of the scale factor and bias:

27

Thus, the relative scale in Equation (25) is also subject to scale errors. This section solves this problem via Equation (18) for two semantic sources with different scales according to Equation (25), but under the condition that the relative scale is erroneous. Nevertheless, we assume that this error is moderate, i.e., |Δγ/γ| ≤ 5%. We then simulate the effect of the inaccurate estimate of Equation (25), as summarized in Table 3. The cases for which the resultant errors differ from the correctly scaled version are highlighted in bold font.

The overall errors increase slightly for some cases in the easy, medium, and hard groups. Nevertheless, these deviations are less than a couple of meters, which should be admissible in the absence of a GNSS or in a GNSS-challenged environment. Experimental results for the easy and medium classes indicate that the accuracy of the proposed algorithm is robust to scale uncertainty as long as a commercial-grade BA is supported. Simply remapping the semantic information from a Cartesian to a polar-coordinated representation can bring immunity to the imprecise scale. However, again in this scaled case, adjusting the parameter N_r to 45 yields heading errors of 178° in both the H1 and H2 cases. This result is due to the high auto-correlation peaks around 180°, as shown in Figure 11. These peak are not random unpredictable errors; rather, they indicate specific symmetry (or reflection) around 180°, where one of the auto-correlation peaks should occur; thus, it is anticipated that identifying such extreme false cases will be feasible if the estimates of the previous step are available in practical applications.

The local ambiguity due to the symmetry and, thus, the recurrence of similar patterns can be resolved by including a broader range of images so that such local patterns are no longer repeated or symmetric; however, the user should be careful about adjusting parameters, as the performance is sensitive to these choices. Utilizing a less conservative value for α, e.g., 0.01, can also prevent the true VSCs from being rejected when the segmentation results are not sufficiently sharp. Again, emphasis is placed on the concept of optimal VSC-based navigation through the optimal choice of parameters. Needless to say, all of the estimates exhibited perfect matching when the exemplary database was used.