college and sickness claim

I was thinking about doing some part time college like 4 hours a week , nothing that includes anything physical just sitting maths and English, I am not sure on the rules i get PIP and ESA , My doctor said it would be good if i could  try getting out of the house more for my over all health,  but i am  worried they might stop my claim or say i am fit for work if i do go college.