Abstract
Contrastive Language-Audio Pretraining (CLAP) models have demonstrated unprecedented performance in various acoustic signal recognition tasks. Fiber-optic-based acoustic recognition is one of the most important downstream tasks and plays a significant role in environmental sensing. Adapting CLAP for fiber-optic acoustic recognition has become an active research area. As a non-conventional acoustic sensor, fiber-optic acoustic recognition presents a challenging, domain-specific, low-shot deployment environment with significant domain shifts due to unique frequency response and noise characteristics. To address these challenges, we propose a support-based adaptation method, CLAP-S, which linearly interpolates a CLAP Adapter with the Support Set, leveraging both implicit knowledge through fine-tuning and explicit knowledge retrieved from memory for cross-domain generalization. Experimental results show that our method delivers competitive performance on both laboratory-recorded fiber-optic ESC-50 datasets and a real-world fiber-optic gunshot-firework dataset. Our research also provides valuable insights for other downstream acoustic recognition tasks.
Highlights
Use the labels twice
The same few labeled samples fine-tune a lightweight adapter and fill a key-value support set. Prediction blends what the model learned with what it can look up.
Is zero-shot knowledge always helpful?
Not under a large domain shift. On fiber audio, adding CLAP's zero-shot text classifier lowers accuracy, and task-aligned embeddings retrieve better than text-aligned ones.
Lab and real-world fiber data
We evaluate on ESC-50 recorded through two fiber sensors and on an outdoor gunshot–firework dataset from telecom cables, which we release with the code.
Method
Support set. Given \(K\)-shot \(N\)-class training audio, the audio encoder produces embeddings \(F_{\text{train}} \in \mathbb{R}^{NK \times C}\) and the labels form one-hot vectors \(L_{\text{train}} \in \mathbb{R}^{NK \times N}\). These are the keys and values of the support set, which stores the new knowledge from the training data.
Implicit + explicit knowledge. A test embedding \(u\) is the query. It is classified by the CLAP head (implicit knowledge in the model weights) and by cross-attention over the support set (explicit knowledge in memory). The two predictions are mixed with weight \(\alpha \in [0, 1]\):
$$p_{\text{final}}(y \mid x) = (1 - \alpha)\, p_{\text{clap}}(y \mid x, u) + \alpha\, p_{\text{support}}(y \mid x, u),$$ $$p_{\text{clap}}(y \mid x, u) = u W_c^{\top},$$ $$p_{\text{support}}(y \mid x, u) = e^{-\beta (1 - u F_{\text{train}}^{\top})} L_{\text{train}}^{\top},$$where \(\beta\) controls sharpness. The embedding \(u\) is either the text-aligned output of the frozen encoder or the task-aligned output of a two-layer MLP adapter fine-tuned on the same labeled data:
$$u_0 = \text{AudioEncoder}(x), \qquad u_f = \text{Adapter}(u_0).$$Two variants. CLAP-S is training-free: it uses \(u = u_0\) and \(\alpha = 1\), so the prediction comes only from the support set. CLAP-S+ uses the task-aligned \(u = u_f\) for both classification and retrieval with \(0 < \alpha < 1\).
Relation to existing methods
| Method | \(p_{\text{clap}}\) | \(p_{\text{support}}\) | Combination | Zero-shot head |
|---|---|---|---|---|
| ZS-CLAP | \(u_0\) | – | Zero-shot (\(\alpha = 0\)) | Yes |
| Tip-Adapter | \(u_0\) | \(u_0\) | Zero-shot + support set | Yes |
| Tip-Adapter-F | \(u_0\) | \(u_f\) | Zero-shot + fine-tuned support set | Yes |
| CLAP-S | – | \(u_0\) | Support set only (\(\alpha = 1\)) | No |
| CLAP-S+ | \(u_f\) | \(u_f\) | Adapter + task-aligned support set | No |
Unlike Tip-Adapter and Treff, CLAP-S+ drops the zero-shot text head and uses the fine-tuned adapter embedding for both the query and the keys.
Datasets
Lab task: ESC-50 is replayed and recorded through a fiber mandrel (FM) and a fiber coil (FC) with a distributed acoustic sensing system, alongside a reference microphone (ECM). Real task: an outdoor gunshot–firework dataset collected from pre-deployed telecom cables with two sensor types (FMO, FCO). It has eight event classes, including gunshots, crackers, cannons, fireworks, car door slams and car alarms.
| Dataset | Classes | Train | Val | Test |
|---|---|---|---|---|
| Microphone (ECM) | 50 | 1400 | 400 | 200 |
| Fiber Mandrel (FM) | 50 | 1400 | 400 | 200 |
| Fiber Coil (FC) | 50 | 1400 | 400 | 200 |
| Fiber Mandrel Outdoor (FMO) | 8 | 647 | 167 | 421 |
| Fiber Coil Outdoor (FCO) | 8 | 406 | 106 | 263 |
The first three are the lab task and the last two the real task. The outdoor gunshot–firework data can be downloaded from the dataset link.
Mel spectrograms of a starter gun and a firework cannon, each recorded by the two outdoor fiber sensors.
Experimental Results
Zero-shot and full-shot adaptation
| Method | Lab task | Real task | Avg. | |||
|---|---|---|---|---|---|---|
| ECM | FM | FC | FMO | FCO | ||
| Zero-shot (prompt template) | ||||||
| [class] | 71.8 | 35.2 | 22.1 | 18.0 | 14.0 | 32.2 |
| this is [class] | 79.9 | 42.9 | 27.4 | 20.0 | 10.0 | 36.0 |
| this is an audio of [class] | 80.7 | 44.3 | 27.2 | 17.0 | 10.0 | 35.8 |
| i can hear the sound of [class] | 77.5 | 40.6 | 27.9 | 16.0 | 16.0 | 35.6 |
| Training-free | ||||||
| Tip-Adapter | 87.0±0.1 | 59.0±0.1 | 39.0±0.1 | 78.8±1.5 | 82.2±1.0 | 69.2 |
| CLAP-S (ours) | 92.0 +5.0 | 61.4 +2.4 | 43.0 +4.0 | 79.0 +0.2 | 83.0 +0.8 | 71.6 +2.4 |
| Training-required | ||||||
| Prompt Tuning | 87.3±1.7 | 46.3±1.8 | 30.2±1.2 | 4.0±4.0 | 5.0±5.2 | 34.6 |
| Adapter | 92.9±1.1 | 68.8±1.0 | 48.8±1.7 | 81.4±0.9 | 90.2±0.7 | 76.4 |
| Treff | 90.3±0.8 | 67.5±0.9 | 46.8±1.0 | 84.2±1.3 | 90.1±1.7 | 75.8 |
| Tip-Adapter-F | 91.0±0.7 | 68.6±1.0 | 47.4±0.8 | 84.6±1.2 | 90.8±0.7 | 76.5 |
| CLAP-S+ (ours) | 94.0 +3.0 | 70.0 +1.4 | 51.0 +3.6 | 87.0 +2.4 | 92.0 +1.2 | 78.8 +2.3 |
Zero-shot CLAP reaches at most 20% on the real task, where classes such as “fountain cannon” are rare in pre-training and wind noise is strong. CLAP-S gains are measured against Tip-Adapter, and CLAP-S+ gains against Tip-Adapter-F. Results are averaged over five runs. Prompt Tuning reaches only 4–5% on the real task, which suggests that maximizing pre-trained knowledge does not help on audio the model has never seen.
Few-shot adaptation
Accuracy from 0 to 24 shots on four datasets. Both CLAP-S and CLAP-S+ improve consistently and stay competitive with the baselines.
Efficiency
| Method | Acc. | Params | Train | Infer. |
|---|---|---|---|---|
| Tip-Adapter | 69.2 | 0 | 5 min | 55 ms |
| CLAP-S | 71.6 | 0 | 4 min | 45 ms |
| Prompt Tuning | 34.6 | 6.14K | 30 min | 266 ms |
| Adapter | 76.4 | 0.52M | 15 min | 96 ms |
| Tip-Adapter-F | 76.5 | 1.43M | 10 min | 70 ms |
| CLAP-S+ | 78.8 | 0.52M | 7 min | 56 ms |
CLAP-S needs no training or extra parameters and is the fastest. CLAP-S+ reaches the best accuracy with fewer parameters and less training time than Tip-Adapter-F.
What Matters Under Domain Shift
| Method | ECM | FM | FC | FMO | FCO | Avg. |
|---|---|---|---|---|---|---|
| Support Set | 92.0 | 61.4 | 43.0 | 79.0 | 83.0 | 71.6 |
| ZS + Support Set | 87.0 | 59.0 | 39.0 | 78.8 | 82.2 | 69.2 |
| ZS + Support Set+ | 91.8 | 58.6 | 39.2 | 25.0 | 20.0 | 46.9 |
| Adapter | 92.9 | 68.8 | 48.8 | 81.4 | 90.2 | 76.4 |
| Adapter + ZS | 94.0 | 68.6 | 48.0 | 82.2 | 85.8 | 75.7 |
| Adapter + Support Set | 93.4 | 71.6 | 50.6 | 84.0 | 88.3 | 77.6 |
| Adapter + Support Set+ (CLAP-S+) | 94.0 | 70.0 | 51.0 | 87.0 | 92.0 | 78.8 |
“Support Set” retrieves with text-aligned embeddings \(u_0\), and “Support Set+” with task-aligned embeddings \(u_f\). ZS is the zero-shot text head.
- Zero-shot knowledge can hurt. Adding the zero-shot head lowers accuracy for both the support set (71.6 → 69.2) and the adapter (76.4 → 75.7). Large domain gaps misalign audio and text, and new acoustic events are hard to describe in language.
- Task-aligned retrieval works better. With the adapter, the task-aligned support set beats the text-aligned one (78.8 vs. 77.6), largely from the real-world FMO and FCO sets.
- Joint training helps. Training one adapter on all five datasets beats training one per dataset (77.4 vs. 76.4 average), acting as a form of data augmentation.