Testing & Validation
80+ experiments with corrected methodology and self-critical audit
Self-Critical Audit
During Phase I we ran an internal audit of our own methodology and found serious problems. Our initial extended-range predictions were producing tautological results — the conclusions we claimed to demonstrate were already embedded in the assumptions. Specifically:
- Detection thresholds were set using signal-present data (oracle), inflating processing gain measurements
- BER tests used as few as 50 bits, making zero-error results statistically meaningless — a 50-bit test with zero errors only bounds BER below 0.06
- SNR was measured after filtering rather than at the microphone, obscuring the actual link budget relationship
- DOA estimates used synthetic angle-plus-noise inputs rather than simulated multi-microphone array signals with real propagation delays
Rather than present unvalidated claims, we discarded the flawed results and rebuilt every measurement with corrected methodology. The re-validated results told a different — and in several cases significantly more optimistic — story. Every number on this site reflects the corrected measurements.
Methodology Corrections
CFAR Detection (No Oracle)
Matched filter threshold set via constant false alarm rate estimation from noise-only statistics — not by looking at signal-present data. Eliminates circular reasoning in processing gain claims.
Result: +23 dB at 200 ms, +29 dB at 500 ms, +31 dB at 1 s
1000-Bit BER with Confidence Intervals
All BER measurements use 1000+ transmitted bits per test point with Wilson score 95% confidence intervals. Previous tests used as few as 50 bits, making zero-error results statistically meaningless.
Result: Threshold confirmed at −10 dB; zero-error results bounded at BER < 0.004 (95% CI)
SNR Defined at Microphone
SNR is measured at the microphone (pre-filtering), connecting the link budget directly to BER performance. Confirmed that filtering provides 0 dB gain within the communication band — the quiet-band advantage is intrinsic.
Result: Mic-level and post-filter SNR identical at 3–5 kHz; link budget uses mic-level threshold
Actual MUSIC DOA Processing
DOA estimates computed from simulated multi-microphone array signals with propagation delays and noise — not from synthetic angle-plus-noise. Full eigendecomposition and spectral scan.
Result: 2.7° mean error (median 0.3°); 0 collisions in integrated Boids test
Validated Results
11 core experiments spanning channel characterization, data rate optimization, swarm coordination, environmental resilience, and full-stack integration. All results obtained with the corrected methodology described above.
Core
Frequency Comparison
Communication at 3–5 kHz successfully decodes at −10 dB threshold (200 ms symbols). Legacy approach at 200 Hz cannot find a threshold at all.
Matched Filter Processing Gain
+23 dB measured processing gain at 200 ms symbol duration, scaling to +31 dB at 1 s. Threshold measured via CFAR (constant false alarm rate), not oracle.
Noise Spectrum Analysis
3–5 kHz band confirmed at −35.9 dB relative to blade-pass frequency region. The advantage is intrinsic to the frequency choice.
Link Budget Validation
Simulation predicts 50–200 m range. Conservatively claiming 10–50 m operational, 100 m ideal, with 6–10 dB derating for real-world unknowns pending hardware validation.
Data Rate
M-ary Modulation
Validated up to 32-ary modulation (5 bits/symbol). Achievable rates: 20 bps (4-ary, 100 ms), 30 bps (8-ary), 40 bps (16-ary). Higher rates possible with parallel channels.
Swarm
Passive Sensing Coordination
DOA + ranging with zero data exchange achieves 90%+ of perfect coordination performance (3.8 m avg separation vs 3.5 m perfect). Adding even modest data closes the gap entirely.
Scalability
Multi-Drone Scaling
TDMA with 12 drones: BER = 0.046, under the 0.05 target. FDMA splits into 3 simultaneous channels. Hybrid approach scales further.
Environmental
Wind Resilience
BER below measurement threshold (< 0.004, 95% CI) at all wind speeds up to 12 m/s (27 mph). Wind noise negligible at operating frequencies.
Doppler Validation
0 dB penalty to 2 m/s, +9 dB at 10 m/s (200 ms symbols). At 30 m/s: BER = 0.028, still operational. At 0 dB SNR: BER below measurement threshold to 15 m/s.
Integration
Full Integration Test
5 drones, full stack (modem + DOA + ranging + flocking) over 50 simulated seconds: zero collisions, 2.7° mean DOA error, 0.29 m range error, stable 2.7 m separation.
Audit
Methodology Corrections
Re-validated all claims with CFAR detection (no oracle), 1000-bit BER tests with Wilson score 95% CI, and actual MUSIC DOA processing on simulated mic array signals.
Statistical Rigor
Wilson Score CI
All BER measurements report 95% Wilson score confidence intervals. Unlike the normal approximation, Wilson intervals remain valid at low error counts and never produce negative lower bounds. A zero-error result on 1000 bits is reported as BER < 0.004, not “zero.”
CFAR Detection
Detection thresholds are set via constant false alarm rate estimation using noise-only statistics. The threshold is chosen before examining signal-present data, eliminating circular reasoning. This is the standard approach in radar/sonar signal processing.
1000-Bit Minimum
Every BER test point transmits at least 1000 bits. Previous tests used as few as 50 bits — far too few for meaningful statistics. With 1000 bits, a zero-error measurement bounds BER below 0.004 at 95% confidence rather than the essentially uninformative bound of 0.06 from 50 bits.
No Oracle Thresholds
No measurement uses knowledge of the transmitted signal to set detection thresholds. Processing gain is measured under realistic conditions where the receiver must distinguish signal from noise without privileged information about what was sent.
Note: All results are software-demonstrated using the Aurora encode/channel/decode pipeline. Monte Carlo testing (10+ independent seeds) was applied to stochastic experiments. The 80+ experiment count includes methodology validation runs, parameter sweeps, and the 11 core experiments reported here. All claims are bounded by corrected statistical methods; no unbounded extrapolations are presented.