Non-intrusive load monitoring (NILM) research has produced a rich body of machine learning approaches for appliance state detection and energy disaggregation. However, the community lacks standardised metrics for evaluating how well models trained on one dataset transfer to another. This paper addresses this gap by analysing existing evaluation metrics, identifying their limitations for cross-dataset transfer, and proposing a framework of transferability metrics grounded in information-theoretic and statistical measures. We demonstrate the framework on multiple residential energy datasets and show how commonly used metrics such as F1 score can be misleading when assessing model transferability. Our findings provide guidance for researchers on selecting appropriate metrics when reporting NILM model performance in multi-dataset settings.