I am not a doctor, nor do I play one on TV. This is my experience, it may or may not apply to you. YMMV
Eight years ago, I broke my left collarbone, and destroyed my rotator cuff. I could barely move my left arm.
The doctor said it would heal naturally. As per medical advice, I was told the sling was for my comfort only. I only used it in public places to keep people from bumping into it.
I started doing everything with my left hand (I am right handed). I used pain as my guide. I moved my arm until just beyond the pain point. and then stopped. Within two weeks, much to the surprise of the doctor, I had full use of my rotator cuff.
On the third week, with medical approval, I started riding an indoor cycle, but I did not lean on the handle bars, and tried to avoid jostling the arm.
Three weeks later I was riding my bicycle again as the collarbone had healed. I was told I did not need physical therapy.
The most important lesson, confirmed with some doctors, is that by appropriately moving my arm, I got the blood flowing to my arm which aided in recovery. Not moving it risks scar tissue that could result in a frozen shoulder.
Everything depends on the nature of the fracture, and your natural healing abilities.