Storm Composite Analysis Walkthrough

Date: 2026-06-16
Working directory: storm_composit_analysis/

This note summarizes the analysis path used to reproduce and extend the Hurricane Spotter wind-speed/quadrant composites, diagnose sampling and quadrant-label issues, compare the new Doppler correction against Jake’s processed Idalia data, and build the later peak-frequency/MSS/ISOMAP diagnostics.

1. Starting Point: Reproduce The Tuned Quadrant Composite

The original target figure was ws_binned_spectra_quad_composit.png, produced by a notebook in Hurricane_spotter. The first task was to find the plotting block that made that exact figure, copy the plotting workflow into a new analysis area, and preserve the formatting, labels, color choices, and spacing.

The working notebook is:

• storm_composit_analysis/wind_speed_binned_spectra_quad_composite.ipynb
• executed copy: storm_composit_analysis/wind_speed_binned_spectra_quad_composite.executed.ipynb

Primary derived datasets:

• storm_composit_analysis/common_frequency_storm_composite_samples.nc
• storm_composit_analysis/common_frequency_spotter_storm_composite_samples.nc
• storm_composit_analysis/frequency_valid_storm_composite_samples.nc

The main Spotter-only common-frequency dataset is:

storm_composit_analysis/common_frequency_spotter_storm_composite_samples.nc  

It contains the corrected spectra interpolated to a shared frequency grid:

freq = np.geomspace(0.01, 0.5, 96)  

The current quadrant composite figures include:

• storm_composit_analysis/spotter_ws_binned_spectra_quad_composit.png
  spotter_ws_binned_spectra_quad_composit.png
• storm_composit_analysis/ian_2022_spotter_ws_binned_spectra_quad_composit.png
  ian_2022_spotter_ws_binned_spectra_quad_composit.png
• storm_composit_analysis/idalia_2023_spotter_ws_binned_spectra_quad_composit.png
  idalia_2023_spotter_ws_binned_spectra_quad_composit_1.png
• storm_composit_analysis/milton_2024_spotter_ws_binned_spectra_quad_composit.png
  milton_2024_spotter_ws_binned_spectra_quad_composit.png
• storm_composit_analysis/ws_binned_spectra_quad_composit_all_storms.png
  ws_binned_spectra_quad_composit_all_storms.png
• storm_composit_analysis/idalia_processed_algebraic_ws_binned_spectra_quad_composit.png
  idalia_processed_algebraic_ws_binned_spectra_quad_composit.png

Later, all active quadrant-style plots were standardized to:

spectra y-limits: 1e-3 to 1e3  
alignment y-limits: -20 to 200 deg  
alignment major ticks: 0, 90, 180 deg  

2. Frequency Support And Patchiness

Patchiness in the spectra was traced to uneven or invalid frequency support. Some observations had mapped intrinsic frequencies that were zero, non-finite, or otherwise not usable at parts of the common frequency grid.

The frequency-valid rule became:

Use only timestamps where frequency_intrinsic_mapped is finite and > 0,  
and energy_density_intrinsic_mapped is finite and > 0, in enough native/source bins.  

This was recorded in the common-frequency dataset metadata as:

frequency_sample_filter =  
"Kept only timestamps with frequency_intrinsic_mapped finite and > 0,  
and energy_density_intrinsic_mapped finite and > 0, in at least 10 native/source frequency bins."  

Audit output:

• storm_composit_analysis/spotter_frequency_support_drop_audit.csv

Conclusion:

The patchiness was not just a plotting artifact. It came from samples whose Doppler-mapped frequency coordinates were not valid across enough of the analysis band. The common-frequency composite dataset now carries the valid-frequency mask and only includes samples with enough usable support.

Additional interpretation:

The important implication is that apparent spectral structure can be generated by sampling geometry alone if frequency support is not tracked explicitly. In the tropical cyclone environment, Doppler remapping can compress or stretch portions of the intrinsic spectrum, causing some observations to contribute unevenly across frequency. The frequency-support audit therefore acts as a quality-control filter on the coordinate transformation itself rather than on the wave physics. This increases confidence that later differences in spectral shape, directional spreading, or high-frequency energy content reflect physical variability rather than interpolation artifacts.

3. Spotter vs MicroSWIFT

Early combined Spotter/MicroSWIFT composites were not coherent enough. The MicroSWIFT and Spotter spectra behaved differently enough that combined binning reduced interpretability.

Decision:

Use Spotter-only composites for the main Ian/Idalia/Milton analysis.  

The active common-frequency dataset for most later work is therefore:

storm_composit_analysis/common_frequency_spotter_storm_composite_samples.nc  

4. Quadrant Labels And Algebraic Reassignment

The quadrant labels were checked against a direct storm-motion coordinate calculation. The diagnostic algebra was:

$$s=\mathbf{r}\cdot \hat{\mathbf{t}}$$ $$c=\mathbf{r}\cdot {\hat{\mathbf{n}}}_R$$

where:

• $\mathbf{r}$ is the center-to-buoy vector
• $\hat{\mathbf{t}}$ is the storm-motion unit vector
• ${\hat{\mathbf{n}}}_R$ is the right-hand normal to storm motion

The quadrant assignment is:

$$\begin{array}{ll}s>0,c>0& :\text{front-right}\\ s>0,c<0& :\text{front-left}\\ s<0,c>0& :\text{back-right}\\ s<0,c<0& :\text{back-left}\end{array}$$

Key outputs:

• storm_composit_analysis/idalia_processed_spotter_quadrant_algebra_summary.csv
• storm_composit_analysis/idalia_processed_spotter_quadrant_algebra_confusion_counts.csv
• storm_composit_analysis/idalia_processed_spotter_quadrant_algebra_mismatches.csv
• storm_composit_analysis/idalia_processed_spotter_algebraic_quadrants.nc
• storm_composit_analysis/idalia_processed_algebraic_ws_binned_spectra_quad_composit.png
  idalia_processed_algebraic_ws_binned_spectra_quad_composit 1.png

Script:

• storm_composit_analysis/check_idalia_processed_quadrants.py
• storm_composit_analysis/make_idalia_processed_algebraic_quadrants.py

Conclusion:

The stored quadrant labels were not always algebraically consistent with the center-to-buoy vector and storm-motion direction. For the robust Idalia comparison, the old label was preserved and a new algebraic label was regenerated.

5. Center Track, Spline, HURDAT2, And Reassigned Observations

To inspect whether quadrant reassignment was driven by center position or heading convention, center-track plots were added.

Key figures:

• storm_composit_analysis/idalia_center_spline_after_translation_vs_hurdat2.png idalia_center_spline_after_translation_vs_hurdat2.png
• storm_composit_analysis/idalia_center_spline_before_translation_vs_hurdat2.png
  idalia_center_spline_before_translation_vs_hurdat2.png
• storm_composit_analysis/idalia_center_spline_after_before_translation_vs_hurdat2.png
  idalia_center_spline_after_before_translation_vs_hurdat2.png
• storm_composit_analysis/idalia_reassigned_observation_locations_with_tracks.png
  idalia_reassigned_observation_locations_with_tracks.png
• storm_composit_analysis/idalia_reassigned_observation_locations_by_quadrant.png
  idalia_reassigned_observation_locations_by_quadrant.png

Script:

• storm_composit_analysis/plot_idalia_center_spline_tracks.py
• storm_composit_analysis/plot_idalia_reassigned_observations_with_tracks.py

Conclusion:

The reassigned-observation map plus center-track overlays made it easier to separate center-location issues from heading-window issues.

6. Heading-Window And Quadrant-Boundary Sensitivity

The heading used for quadrant assignment was tested with several smoothing windows:

0.5 hr, 1 hr, 2 hr, 3 hr  

Samples were flagged if:

1. The quadrant changed across heading windows.
2. The sample was close to a quadrant boundary.

Boundary proximity was based on the angular distance from the nearest quadrant line. The practical ambiguous margin was about 15 degrees.

Key outputs:

• storm_composit_analysis/idalia_processed_heading_window_quadrant_sensitivity.csv
• storm_composit_analysis/idalia_processed_quadrant_suspicious_points.csv
• storm_composit_analysis/idalia_processed_quadrant_suspicious_arrows.png
  idalia_processed_quadrant_suspicious_arrows.png
• storm_composit_analysis/all_storms_quadrant_window_sensitivity_diagnostics.csv
• storm_composit_analysis/all_storms_quadrant_window_sensitivity_summary.csv
• storm_composit_analysis/all_storms_quadrant_window_sensitivity_summary.png
  all_storms_quadrant_window_sensitivity_summary.png
• storm_composit_analysis/all_storms_quadrant_sensitivity_stacked_bupu.png
  all_storms_quadrant_sensitivity_stacked_bupu_2.png

Scripts:

• storm_composit_analysis/diagnose_all_storm_quadrant_sensitivity.py
• storm_composit_analysis/plot_quadrant_sensitivity_stacked_bupu.py

All-storm summary:

Ian:    boundary 29.50%, heading-sensitive 25.18%, both 15.35%, stable 60.67%  
Idalia: boundary 42.28%, heading-sensitive 25.16%, both 15.01%, stable 47.57%  
Milton: boundary 25.10%, heading-sensitive 27.78%, both 12.93%, stable 60.06%  
All:    boundary 30.25%, heading-sensitive 26.58%, both 13.96%, stable 57.14%  

Conclusion:

A nontrivial fraction of observations are close to quadrant boundaries or sensitive to heading-window choice. Those samples should be flagged or downweighted for quadrant composites rather than treated as hard labels.

7. Alignment Calculation

The alignment panels use absolute wind-wave alignment:

$$\Delta \theta (f)=\left|\left({\theta }_{\mathrm{wave}}(f)-{\theta }_{\mathrm{wind}}+180^{\circ }\right)\mathrm{mod}360^{\circ }-180^{\circ }\right|$$

The suspicious low-frequency alignment behavior was addressed by not plotting alignment below:

$$f=0.04\mathrm{Hz}$$

This removes the unrealistically noisy lowest-frequency alignment points while retaining the rest of the profile.

Conclusion:

The alignment issue was localized to the lowest-frequency tail. The plotting rule is now to mask alignment below 0.04 Hz.

8. Jake Processed Data vs Our Doppler Correction

The major comparison was whether the new Doppler correction changed the spectrum in a way that was different from Jake’s processed Idalia dataset, and whether the common-frequency interpolation was responsible.

Variance conservation was checked with:

$$m_0=\int E(f)df$$ $$H_s=4\sqrt{m_0}$$

Matched-observation outputs:

• storm_composit_analysis/idalia_matched_spectral_variance_conservation.csv
• storm_composit_analysis/idalia_spectral_variance_conservation_summary.csv
• storm_composit_analysis/idalia_matched_peak_frequency_shift_diagnostics.csv
• storm_composit_analysis/idalia_one_matched_observation_spectral_variance_curves.png
  idalia_one_matched_observation_spectral_variance_curves.png
• storm_composit_analysis/idalia_peak_frequency_shift_summary.png
  idalia_peak_frequency_shift_summary.png

Script:

• storm_composit_analysis/diagnose_idalia_spectral_variance_conservation.py

Most diagnostic single figure:

storm_composit_analysis/idalia_one_matched_observation_spectral_variance_curves.png  

It overlays:

• raw Spotter E(f)
• Jake/processed Idalia dataset E(f)
• our corrected mapped native-grid E(f)
• our corrected common-grid E(f)

Aggregate peak-shift figure:

storm_composit_analysis/idalia_peak_frequency_shift_summary.png  

Important result:

common-grid / pre-common median peak ratio = 1.000  
pre-common / Jake processed median peak ratio ~ 1.023  

Conclusion:

The common-frequency interpolation is not the main source of the peak-frequency shift. The shift is already present in the corrected mapped native-grid spectra, so the difference is associated with the Doppler correction/remapping step relative to Jake’s processed dataset.

9. Rebinning Peak Frequency And MSS For Idalia

The first attempt selected Idalia spectra using stored ISOMAP fp_bin=8 and mss_bin=12. Some spectra visually had very different peak frequencies because those labels came from the ISOMAP feature table rather than the exact plotted common-grid spectra.

Old diagnostic figure:

• storm_composit_analysis/idalia_fp8_mss12_spectra_alignment_colored_by_directional_spread.png
  idalia_fp8_mss12_spectra_alignment_colored_by_directional_spread.png

To fix this, Idalia spectra were rebinned using:

1. Actual plotted common-grid peak frequency:

$$f_{p,\mathrm{common}}=\underset{f}{\mathrm{argmax}}{E}_{\mathrm{common}}(f)$$

2. Corrected mean square slope:

mean_square_slope_intrinsic_new  

Selected bin:

common-grid peak: 0.14-0.17 Hz  
MSS: 0.008-0.010  
n = 32  

Outputs:

• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_spectra_alignment_colored_by_directional_spread.png
  idalia_corrected_rebinned_peak_mss_spectra_alignment_colored_by_directional_spread.png
• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_spectra_alignment_samples.csv
• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_metadata.csv
• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_category_counts.csv

Script:

• storm_composit_analysis/plot_idalia_rebinned_peak_mss_directional_spread_profiles.py

Selected peak frequencies:

0.145363 Hz: 8  
0.151474 Hz: 8  
0.157841 Hz: 7  
0.164477 Hz: 9  

Conclusion:

The rebinned selection removes the off-bin spectra around roughly 0.08 Hz and 0.28-0.33 Hz. The plotted spectra are now coherent with the stated peak-frequency/MSS category.

10. Directional Spread And High-Frequency Quartile Medians

Using the coherent Idalia bin above, spectra were split into lower and upper quartiles by several metrics. Only the lower and upper quartile medians were plotted; the middle 50% was omitted.

Directional Spread

Figure:

• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_directional_spread_quartile_medians.png
  idalia_corrected_rebinned_peak_mss_directional_spread_quartile_medians.png

Table:

• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_directional_spread_quartile_medians.csv

Script:

• storm_composit_analysis/plot_idalia_rebinned_peak_mss_directional_spread_quartile_medians.py

Split:

lower quartile: n=8,  DS = 16.27-18.43 deg  
middle 50%:     n=16, DS = 18.46-33.43 deg  
upper quartile: n=8,  DS = 33.86-51.22 deg  

High-Frequency Directional Spread

Figure:

• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_hf_directional_spread_quartile_medians.png
  idalia_corrected_rebinned_peak_mss_hf_directional_spread_quartile_medians.png

Table:

• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_hf_directional_spread_quartile_medians.csv

High-Frequency Tail Energy Fraction

Figure:

• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_hf_tail_fraction_quartile_medians.png
  idalia_corrected_rebinned_peak_mss_hf_tail_fraction_quartile_medians.png

Table:

• storm_composit_analysis/idalia_corrected_rebinned_peak_mss_hf_tail_fraction_quartile_medians.csv

Script:

• storm_composit_analysis/plot_idalia_rebinned_peak_mss_hf_metric_quartile_medians.py

HF splits:

HF directional spread  
lower quartile: n=8,  34.05-37.69 deg  
middle 50%:     n=16, 39.30-52.01 deg  
upper quartile: n=8,  52.35-67.99 deg  
  
HF tail energy fraction  
lower quartile: n=8,  0.0339-0.1399  
middle 50%:     n=16, 0.1589-0.4227  
upper quartile: n=8,  0.4821-0.8418  

Conclusion:

Within a fixed peak-frequency/MSS bin, high-frequency metrics separate meaningful median spectral shapes. The high-frequency tail fraction especially separates spectra with different low-frequency-to-tail energy distributions.

Additional interpretation:

This result is important because the spectra were intentionally selected to occupy nearly the same bulk sea-state space. If spectra with similar peak frequency and MSS still exhibit systematically different high-frequency structure, then peak frequency and MSS alone are insufficient descriptors of the wave field. The separation suggests that organization of the short-wave tail contains information that is largely independent of traditional bulk parameters. This observation provides part of the physical motivation for the later ISOMAP-sheet interpretation and for considering high-frequency directional organization as a potentially important control on air-sea momentum exchange.

11. ISOMAP Lower vs Upper Sheet Median Spectra

The ISOMAP sheet labels were used to compare lower-sheet and upper-sheet median spectra. The plotted spectra still came from the corrected processed common-frequency dataset.

Figures:

• storm_composit_analysis/all_storms_isomap_lower_upper_sheet_median_spectra_alignment.png
  all_storms_isomap_lower_upper_sheet_median_spectra_alignment.png
• storm_composit_analysis/idalia_isomap_lower_upper_sheet_median_spectra_alignment.png
  idalia_isomap_lower_upper_sheet_median_spectra_alignment.png

Tables:

• storm_composit_analysis/isomap_lower_upper_sheet_median_spectra_alignment_samples.csv
• storm_composit_analysis/isomap_lower_upper_sheet_median_spectra_alignment_summary.csv

Script:

• storm_composit_analysis/plot_isomap_sheet_median_spectra_alignment.py

Important matching fix:

The source_sample mapping in isomap_sheet_labels_for_dynamics.nc was stale or misaligned for Milton. Direct mapping through source_sample assigned Milton labels to incorrect sample ids. The final script therefore matches ISOMAP labels to corrected spectra using:

storm + trajectory + time  

Counts:

Ian/Idalia/Milton  
lower: 518  
upper: 125  
  
Idalia only  
lower: 148  
upper: 34  

Conclusion:

The lower/upper ISOMAP sheet separation is visible in the median spectra and alignment profiles. The time/trajectory matching is necessary for any analysis involving Milton labels.

Additional interpretation:

The existence of two distinguishable median spectral structures after dimensionality reduction is more significant than the clustering itself. The key observation is that the sheets appear to occupy overlapping regions of peak-frequency/MSS space while maintaining differences in directional organization and high-frequency energy distribution. In practical terms, this means that two sea states with similar bulk characteristics can reside on different regions of the manifold and exhibit different short-wave behavior. The evidence is suggestive rather than definitive, but it remains one of the strongest indications that the dataset contains structure beyond a simple one-parameter progression with wind speed.

12. Peak Frequency vs MSS Scatter Diagnostics

Scatter plots were made for Ian, Idalia, and Milton using corrected spectral diagnostics. Axes were:

x = peak_frequency  
y = MSS  

Color variables:

1. high-frequency alignment:

tail_abs_peak_hf_offset_deg  

2. high-frequency directional spread:

tail_DS_hf  

3. high-frequency energy fraction:

spectrum_E_hf_fraction  

Figures:

• storm_composit_analysis/ian_idalia_milton_peak_frequency_mss_hf_diagnostics_scatter.png
  ian_idalia_milton_peak_frequency_mss_hf_diagnostics_scatter.png
• storm_composit_analysis/ian_idalia_milton_peak_frequency_mss_colored_by_hf_alignment.png
  ian_idalia_milton_peak_frequency_mss_colored_by_hf_alignment.png
• storm_composit_analysis/ian_idalia_milton_peak_frequency_mss_colored_by_hf_directional_spread.png
  ian_idalia_milton_peak_frequency_mss_colored_by_hf_directional_spread.png
• storm_composit_analysis/ian_idalia_milton_peak_frequency_mss_colored_by_hf_tail_energy_fraction.png
  ian_idalia_milton_peak_frequency_mss_colored_by_hf_tail_energy_fraction.png

Tables:

• storm_composit_analysis/ian_idalia_milton_peak_frequency_mss_hf_diagnostics_scatter_data.csv
• storm_composit_analysis/ian_idalia_milton_peak_frequency_mss_hf_diagnostics_summary.csv

Script:

• storm_composit_analysis/plot_peak_frequency_mss_hf_diagnostic_scatters.py

Counts:

Ian:     432  
Idalia:  461  
Milton: 1404  
Total:   2297  

Conclusion:

High-frequency energy fraction increases strongly with peak frequency and decreases with MSS in the sampled corrected diagnostics. The storm marker shapes make clear that Milton dominates the population, so any storm-combined trend should be checked against per-storm subsets before physical interpretation.

Additional interpretation:

The peak-frequency dependence is expected because increasing peak frequency shifts spectral energy toward shorter waves, naturally increasing the fraction of energy that falls within the high-frequency integration band. The MSS relationship is more nuanced. MSS integrates contributions across a broad range of scales and therefore does not uniquely diagnose how energy is partitioned within the tail. Consequently, the apparent MSS trend should not be interpreted as direct evidence that steeper seas always possess larger or smaller high-frequency fractions. Instead, it indicates that MSS and tail organization are related but non-equivalent descriptors of spectral structure.

13. $R_{hf}$ Relationships

The high-frequency energy fraction is treated as:

$$R_{hf}=\frac{{\int }_{0.20}^{0.50}E(f)df}{{\int }_{0.05}^{0.50}E(f)df}$$

Two follow-up scatter relationships were plotted:

1. (R_{hf}) vs peak frequency, colored by MSS
2. (R_{hf}) vs MSS, colored by peak frequency

Final combined figure:

• storm_composit_analysis/ian_idalia_milton_r_hf_vs_peak_mss_panel_colorbars.png
  ian_idalia_milton_r_hf_vs_peak_mss_panel_colorbars.png

Individual figures:

• storm_composit_analysis/ian_idalia_milton_r_hf_vs_peak_frequency_colored_by_mss.png
  ian_idalia_milton_r_hf_vs_peak_frequency_colored_by_mss.png
• storm_composit_analysis/ian_idalia_milton_hf_energy_fraction_vs_mss_colored_by_peak_frequency.png
  ian_idalia_milton_hf_energy_fraction_vs_mss_colored_by_peak_frequency.png

Script:

• storm_composit_analysis/plot_hf_energy_fraction_vs_peak_mss.py

Conclusion:

This pair of plots is useful for separating two effects:

• (R_{hf}) increases with peak frequency.
• At a fixed or similar peak-frequency range, MSS helps identify whether high-frequency energy fraction is simply tied to slope magnitude or is reflecting spectral redistribution.

14. Current Practical Rules

The following rules emerged from the analysis and should be carried forward:

1. Use Spotter-only data for the main Ian/Idalia/Milton composites unless there is a specific reason to compare MicroSWIFT separately.
2. Only include spectra with finite, positive mapped frequencies and enough valid positive-energy bins.
3. Mask alignment below 0.04 Hz.
4. Use algebraic quadrant labels from dot products when robust quadrant assignment matters.
5. Treat samples near quadrant boundaries or sensitive to heading-window choice as ambiguous.
6. For Milton ISOMAP labels, do not rely on source_sample; match by storm + trajectory + time.
7. When selecting spectra by peak-frequency/MSS bins for plotting, bin using the actual spectra being plotted, not stale or upstream labels from a different grid.
8. The common-grid interpolation conserves the relevant spectral shape well enough for these diagnostics; the Jake-vs-new shift is upstream of the common-grid interpolation.

15. Suggested Next Steps

1. Rebuild any remaining sheet or feature-bin plots using the storm + trajectory + time ISOMAP label match.
2. Add an explicit peak_frequency_common_grid and mss_corrected_bin variable to the derived common-frequency NetCDF so plot scripts do not have to recompute them independently.
3. Add a quadrant_ambiguous flag to composite inputs using heading-window sensitivity and boundary distance.
4. Run the Jake-vs-new Doppler comparison for more than one representative observation, using the same four-curve format as idalia_one_matched_observation_spectral_variance_curves.png.
5. For physical interpretation of (R_{hf}), repeat the scatter plots storm-by-storm or use equal storm weighting so Milton does not dominate the combined trend.