I used to have this problem. Finally I realized that what my husband and "many people" might consider to be fun, was not only not fun, but stressful for me. My husband used to want to go camping and I hated it. There was no escape from the sun, it rained and the tent got wet, I was exhausted. I really came to dread vacation. Finally through help from friends, I let him know that we needed to plan a vacation on both of our needs or that he'd have to go camping with friends. Now it wasn't easy, and a risk (I wasn't sure if he could handle a sick spouse), but eventually he came to understand that we had different ideas about vacation.
I would let your husband know that : 1. You will go along for the his sake and the kids sake, but that vacation
is hard for you. 2. That although it may seem that this vacation will be good for you, it will really be hard for you and your illness will not go away because you are away and spending time in the sun (in fact the heat and sun aggravate many chronic illnesses).
You may want to agree to go, but let him know what you WILL need (i.e., a nap in the afternoon, shade, time for yourself) and plan a vacation based on what you would like (i.e., a weekend stay in a hotel where you can rest, order room service, soak in a hot tub etc...). You can't live a lie.
I know it's hard. Now when I go away though it is always stressful, I make sure that I get the rest I need and bow out of activities that are harmful to my health.