Holidays - Afraid to even consider taking one

So my wife and I haven't had a holiday in well over a decade, she suggested maybe we take one?

In the past I would have been the driving force, however the idea is causing me serious crushing anxiety - reason being that part of my LCWRA award is down to my mental health (I really don't go out much as my mental health is all over the place and trying to act "normal" is frankly utterly exhausting, even when my wife is with me its a struggle and I can't reliably do it which is so frustrating). I also suffer from chronic pain and exhaustion. My claim has my conditions down as variable which they are (and thats an understatement) some days are better than others, often it can change during the day even, which is beyond frustrating and demoralising.

I'm worried that should we book a holiday (she was looking at a quiet location in Spain where we could have a house or self contained apartment to ourselves to avoid stress as I struggle to be around people and its cheaper to go to Spain than take a holiday in the UK by a sizeable sum of money) that the DWP will very very much use it against me in future - "your mental health can't be that bad if you were able to take a holiday" "I thought you never went out" "Well if you could manage that you are clearly fit for work and we are going to prosecute you for benefits fraud" (my anxiety is running riot with that concept, mainly due to the feeling that the DWP is constantly looking for ways to catch you out and their view is 'You are guilty of something and we will find something') - Am I being paranoid or am I leaving myself open to a serious headache in future by even considering this?