That seriously annoys me when people say things like that.
I used to do dressage, eventing and jumping. I took a really nasty fall while jumping - the horse's feet went out from under him on takeoff, and he and the jump landed on top of me. It was lights out - for a couple hours. Smashed my helmet to smithereens. If the ground had not been soft, I would have been completely crushed through my torso and killed. As it was, the ground sunk and I was protected. No broken bones. Extremely fortunate. But, I have complete loss of memory to this day and still have brain-related repercussions to this day that I have learned to compensate and deal with.
But the point is, I could no longer jump after this. Simply put, another major bash to the head could possibly be the end for me. So, I switched purely to dressage. Yes you can still fall riding in dressage, but the likelihood of a really serious body-smashing fall is much, much less likely than jumping or eventing.
Don't allow anyone to pressure you into doing something you are not sure your body can handle.
A break in 4 places - what a serious injury! You are so lucky you did not bleed out! This will indeed take a very, very long time to heal and not just the bones, but also the ligaments and tendons that were stretched all to wack - and these are what will take the longest actually.
I need to ask: Have you had your bone density tested? You stated it was a "wussy" fall... I don't know how old you are, but if you have poor mineral densities in your bones, it could contribute to serious breaks. Although that being said, an awkward angle can cause all kinds of crazy things to happen. I watched a hockey player slide relatively gently into the board, seemingly a situation that would result in little consequence, until we watched in horrified silence as his leg ribboned into positions that no leg is supposed to be in. He was out for a year and it was 2 years before he was fully right.
Something that will help your bones in this serious time of healing and remodeling is upping your intake of vitamin A, C, D, magnesium, phosphorous and calcium. Usually, an "osteoporosis" multi-vitamin is a good one to take.
I would say, let's set your first goal to be walking again with progressive improvement in the stiffness and pain. When you're allowed to, go to physical therapy religiously and do all the mobility exercises at home without fail. They might hurt, but you must train the joints, ligaments, and tendons mobility again. For the first 4 months walking after the cast is off, be very careful as your bones will be soft from lack of use and prone to refracture.
Mostly, take care of your body and only do what you are comfortable with doing! Take care and keep your chin up. One day at a time! You'll get through this!