CLAP-S: Support Set Based Adaptation for Downstream Fiber-optic Acoustic Recognition

Jingchen Sun2,†, Shaobo Han1, Wataru Kohno1, Changyou Chen2
1NEC Laboratories America, Inc. 2University at Buffalo, SUNY
ICASSP 2025
†Work done during an internship at NEC Laboratories America
Pipeline of CLAP-S+: a test audio passes through the frozen audio encoder and a fine-tuned adapter; the embedding attends over a support set of labeled fiber audio (explicit knowledge) and is also classified against text embeddings (implicit knowledge); the two predictions are mixed with weight alpha.

Pipeline of CLAP-S+. A test sample goes through the frozen audio encoder and a fine-tuned adapter. Its embedding attends over the keys of a support set of labeled fiber audio, and the attention weights pick up the support labels as explicit knowledge. The final prediction linearly interpolates this with the implicit knowledge of the fine-tuned CLAP classifier.

Abstract

Contrastive Language-Audio Pretraining (CLAP) models have demonstrated unprecedented performance in various acoustic signal recognition tasks. Fiber-optic-based acoustic recognition is one of the most important downstream tasks and plays a significant role in environmental sensing. Adapting CLAP for fiber-optic acoustic recognition has become an active research area. As a non-conventional acoustic sensor, fiber-optic acoustic recognition presents a challenging, domain-specific, low-shot deployment environment with significant domain shifts due to unique frequency response and noise characteristics. To address these challenges, we propose a support-based adaptation method, CLAP-S, which linearly interpolates a CLAP Adapter with the Support Set, leveraging both implicit knowledge through fine-tuning and explicit knowledge retrieved from memory for cross-domain generalization. Experimental results show that our method delivers competitive performance on both laboratory-recorded fiber-optic ESC-50 datasets and a real-world fiber-optic gunshot-firework dataset. Our research also provides valuable insights for other downstream acoustic recognition tasks.

Highlights

Use the labels twice

The same few labeled samples fine-tune a lightweight adapter and fill a key-value support set. Prediction blends what the model learned with what it can look up.

Is zero-shot knowledge always helpful?

Not under a large domain shift. On fiber audio, adding CLAP's zero-shot text classifier lowers accuracy, and task-aligned embeddings retrieve better than text-aligned ones.

Lab and real-world fiber data

We evaluate on ESC-50 recorded through two fiber sensors and on an outdoor gunshot–firework dataset from telecom cables, which we release with the code.

Method

Support set. Given \(K\)-shot \(N\)-class training audio, the audio encoder produces embeddings \(F_{\text{train}} \in \mathbb{R}^{NK \times C}\) and the labels form one-hot vectors \(L_{\text{train}} \in \mathbb{R}^{NK \times N}\). These are the keys and values of the support set, which stores the new knowledge from the training data.

Implicit + explicit knowledge. A test embedding \(u\) is the query. It is classified by the CLAP head (implicit knowledge in the model weights) and by cross-attention over the support set (explicit knowledge in memory). The two predictions are mixed with weight \(\alpha \in [0, 1]\):

$$p_{\text{final}}(y \mid x) = (1 - \alpha)\, p_{\text{clap}}(y \mid x, u) + \alpha\, p_{\text{support}}(y \mid x, u),$$ $$p_{\text{clap}}(y \mid x, u) = u W_c^{\top},$$ $$p_{\text{support}}(y \mid x, u) = e^{-\beta (1 - u F_{\text{train}}^{\top})} L_{\text{train}}^{\top},$$

where \(\beta\) controls sharpness. The embedding \(u\) is either the text-aligned output of the frozen encoder or the task-aligned output of a two-layer MLP adapter fine-tuned on the same labeled data:

$$u_0 = \text{AudioEncoder}(x), \qquad u_f = \text{Adapter}(u_0).$$

Two variants. CLAP-S is training-free: it uses \(u = u_0\) and \(\alpha = 1\), so the prediction comes only from the support set. CLAP-S+ uses the task-aligned \(u = u_f\) for both classification and retrieval with \(0 < \alpha < 1\).

Relation to existing methods

Method\(p_{\text{clap}}\)\(p_{\text{support}}\)CombinationZero-shot head
ZS-CLAP\(u_0\)–Zero-shot (\(\alpha = 0\))Yes
Tip-Adapter\(u_0\)\(u_0\)Zero-shot + support setYes
Tip-Adapter-F\(u_0\)\(u_f\)Zero-shot + fine-tuned support setYes
CLAP-S–\(u_0\)Support set only (\(\alpha = 1\))No
CLAP-S+\(u_f\)\(u_f\)Adapter + task-aligned support setNo

Unlike Tip-Adapter and Treff, CLAP-S+ drops the zero-shot text head and uses the fine-tuned adapter embedding for both the query and the keys.

Datasets

Lab task: ESC-50 is replayed and recorded through a fiber mandrel (FM) and a fiber coil (FC) with a distributed acoustic sensing system, alongside a reference microphone (ECM). Real task: an outdoor gunshot–firework dataset collected from pre-deployed telecom cables with two sensor types (FMO, FCO). It has eight event classes, including gunshots, crackers, cannons, fireworks, car door slams and car alarms.

DatasetClassesTrainValTest
Microphone (ECM)501400400200
Fiber Mandrel (FM)501400400200
Fiber Coil (FC)501400400200
Fiber Mandrel Outdoor (FMO)8647167421
Fiber Coil Outdoor (FCO)8406106263

The first three are the lab task and the last two the real task. The outdoor gunshot–firework data can be downloaded from the dataset link.

Mel spectrograms of a starter gun and a firework cannon recorded by the fiber mandrel and fiber coil outdoor sensors

Mel spectrograms of a starter gun and a firework cannon, each recorded by the two outdoor fiber sensors.

Experimental Results

Zero-shot and full-shot adaptation

Method Lab taskReal taskAvg.
ECMFMFCFMOFCO
Zero-shot (prompt template)
[class]71.835.222.118.014.032.2
this is [class]79.942.927.420.010.036.0
this is an audio of [class]80.744.327.217.010.035.8
i can hear the sound of [class]77.540.627.916.016.035.6
Training-free
Tip-Adapter87.0±0.159.0±0.139.0±0.178.8±1.582.2±1.069.2
CLAP-S (ours)92.0 +5.061.4 +2.443.0 +4.079.0 +0.283.0 +0.871.6 +2.4
Training-required
Prompt Tuning87.3±1.746.3±1.830.2±1.24.0±4.05.0±5.234.6
Adapter92.9±1.168.8±1.048.8±1.781.4±0.990.2±0.776.4
Treff90.3±0.867.5±0.946.8±1.084.2±1.390.1±1.775.8
Tip-Adapter-F91.0±0.768.6±1.047.4±0.884.6±1.290.8±0.776.5
CLAP-S+ (ours)94.0 +3.070.0 +1.451.0 +3.687.0 +2.492.0 +1.278.8 +2.3

Zero-shot CLAP reaches at most 20% on the real task, where classes such as “fountain cannon” are rare in pre-training and wind noise is strong. CLAP-S gains are measured against Tip-Adapter, and CLAP-S+ gains against Tip-Adapter-F. Results are averaged over five runs. Prompt Tuning reaches only 4–5% on the real task, which suggests that maximizing pre-trained knowledge does not help on audio the model has never seen.

Few-shot adaptation

Few-shot accuracy from 0 to 24 shots for Tip-Adapter, CLAP-S, Tip-Adapter-F and CLAP-S+ on four datasets

Accuracy from 0 to 24 shots on four datasets. Both CLAP-S and CLAP-S+ improve consistently and stay competitive with the baselines.

Efficiency

MethodAcc.ParamsTrainInfer.
Tip-Adapter69.205 min55 ms
CLAP-S71.604 min45 ms
Prompt Tuning34.66.14K30 min266 ms
Adapter76.40.52M15 min96 ms
Tip-Adapter-F76.51.43M10 min70 ms
CLAP-S+78.80.52M7 min56 ms

CLAP-S needs no training or extra parameters and is the fastest. CLAP-S+ reaches the best accuracy with fewer parameters and less training time than Tip-Adapter-F.

What Matters Under Domain Shift

MethodECMFMFCFMOFCOAvg.
Support Set92.061.443.079.083.071.6
ZS + Support Set87.059.039.078.882.269.2
ZS + Support Set+91.858.639.225.020.046.9
Adapter92.968.848.881.490.276.4
Adapter + ZS94.068.648.082.285.875.7
Adapter + Support Set93.471.650.684.088.377.6
Adapter + Support Set+ (CLAP-S+)94.070.051.087.092.078.8

“Support Set” retrieves with text-aligned embeddings \(u_0\), and “Support Set+” with task-aligned embeddings \(u_f\). ZS is the zero-shot text head.

  • Zero-shot knowledge can hurt. Adding the zero-shot head lowers accuracy for both the support set (71.6 → 69.2) and the adapter (76.4 → 75.7). Large domain gaps misalign audio and text, and new acoustic events are hard to describe in language.
  • Task-aligned retrieval works better. With the adapter, the task-aligned support set beats the text-aligned one (78.8 vs. 77.6), largely from the real-world FMO and FCO sets.
  • Joint training helps. Training one adapter on all five datasets beats training one per dataset (77.4 vs. 76.4 average), acting as a form of data augmentation.

BibTeX

@inproceedings{sun2025clap,
  title={CLAP-S: Support Set Based Adaptation for Downstream Fiber-optic Acoustic Recognition},
  author={Sun, Jingchen and Han, Shaobo and Kohno, Wataru and Chen, Changyou},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2025},
  doi={10.1109/ICASSP49660.2025.10887829}
}