I know exactly what you mean!
I killed my first deer pulling about 40 lbs, with a LOT cruder bow than the cheapest thing available today, there's no comparison. What are you pulling now? Don't feel like you got to keep up with the Jones's to shoot well, you'll continue to outgrow your current set-up over the next few years, but doesnt matter where it is you end up, if it's at 53lbs, then that's fine, if it's 75, then that's cool too.