Struggling with my young adult son

My son, 20 yrs old, has an intense fear/anxiety of engaging with anyone other than immediate family and his older brothers 2 friends (older brother is 21). Getting him to attend health appointments, or appointments of any kind, is extremely difficult, and 90% of the time I end up having to cancel last minute and reschedule. When I'm working or his brothers friends have returned to uni, then my son remains at home all day feeling unable to leave the house, and says he doesn't like being on his own. When I'm not working, I try to get him out and about. When his brothers friends are home they go to the gym, and bowling. I step in to accompanying my son to the gym when they're at uni. My parents used to help me to support my son, but they are in their late late 70's with declining physical/mental health and unable to help very much now. He has been out of education since the age of 10. He has an EHCP, and engaged via WhatsApp messages and emails with one placement, but they withdrew the placement because my son refused to engage face to face online or in person. He had the opportunity for a support worker through his EHCP, but refused to engage with them. Social services have said they can't help unless my son wants their help and support, which he's made clear he doesn't want, saying he's fine. He had a girlfriend who he met via SnapChat last year, their messages developed into facetime chats after about 6 months, then she said she'd like to come to visit. She lived 8 hours drive away, but would come to stay for 3 or 4 weeks every few months or so. They were together just short of a year, then she ended things having met someone else via SnapChat. My son was distraught and is very low and depressed, we're back where we were about 3 or 4 years ago. As I mentioned above he says he doesn't like being left on his own, he said it was okay when his girlfriend was here, but now shes not it feels lonely. He finds it very difficult to deal with change and to let go of things, He has ADHD, is Autistic, dyslexic and has irlens syndrome, sensory difficulties and anxiety.

I'm feeling helpless. I was able to be at home to support him, but with the rise in the cost of living, I've had to increase my hours at work, so I'm not at home to support him, and I'm watching my son regress.

I need to work to keep us afloat financially, I was on carers allowance and working part time, but it got to the point where it was no longer enough money financially (my rent is £1000 alone). My son is in receipt of PIP, and I have applied for Universal Credit for him, and awaiting the outcome. I'm struggling not just with feeling helpless, but with feeling guilty for not being around to support my son. I don't want him to end up like my brother who lives with my parents (he's 47), left education at 16 having struggled all the way through with learning difficulties and anxiety. He no lives in his bedroom, has no friends, no job, no money/income and relies solely on my parents. I can't and won't let that happen, but with working 30 hours a week, I'm struggling to work out how best I can be here for and support my son.

Just needed to talk really, I don't really have anyone that understands, so thanks for listening x