Possible, yes, but exception rather than rule, and for the most obvious of reasons: generally people who become and stay fit are attentive to their health, and a focus on health may be the best bet there is for lasting weight control.
Also you can be lean and unfit.
And the opposite can be true; some people who look thin may be unfit, and may also have an excess of fat around the middle which can place them at high risk for chronic disease. These people are the “lean obese,” looking lean, but having the metabolic profile of the overweight. There is increasing recognition that waist circumference more reliably captures the health risks of body fat than the prevailing body-mass index (BMI).
So the general rule is, exercise makes you fit.