Perhaps it’s not exactly equivalent since this is an LLM, but from what I’ve learnt in my undergrad machine learning course, shouldn’t the test data be separate from the training data?

The train-test (or train-validate-test) split was one of the first few things we learnt to do.

Otherwise, the model can easily get a 100% accuracy (or whatever relevant metric) simply by regurgitating training data, which looks like the case here.