Physical exercise is something we do to maintain our health, or otherwise train our bodies for something. So it isn’t an activity that is an end in and of itself, but is rather something done for the sake of some other end.
So let’s put on our philosopher caps and examine the ends of exercise.
Health: Health is like wealth. Both of them are necessary as a means to living a decent life, so they should only be pursued as a means. To not have to worry about whether you’ll eat tomorrow is a blessing, just as not having cancer is. But the more of each you want to pursue, the more burdens/responsibilities you have regarding it. We have to be careful of not leaving behind one set of chains for another one. Now, since health and wealth are means, the level of responsibility we should take should be commensurate to the sort of life we want to live. If you have excess wealth, give some of it away; if you have excess health, well, you can’t really donate that to charity, but you could donate some of your time spent exercising to helping a charity or spending more time with family. What counts as excess in both cases is, of course, relative to the life; what is excess time for one person might be perfectly suited for an Olympic athelete, etc.
Other stuff: Building on what I said above, this is dependent on whether the other stuff is worth living. Exercising so that you can be a supermodel? Not that great. Exercising so that you can be a great tennis player, or so that you can serve in the army are a bit better.
The best? Exercising so that you can fight in dancing giant robots: