I'm going to start by changing the rear drums and upgrading to better brake shoes... The drums are the only thing that was not changed and the shoes are the cheaper set... Also going to look into stainless braided brake hoses,I'm using new rubber hoses now... If all that fails, then it's time for rear disc brakes...
Your pedal feel is from the front brakes. The rear brakes only work in extreme situations and really only are a supplement to the front. When you pump the brakes, does the pedal feel improve ?
Then look at changing the front brake hoses, that's really the only part that changing will give a better pedal feel(if they're swelling from the pressure, if they're new, they shouldn't swell, the rubber hoses are also steel braided), that or maybe a larger bore master cylinder. The front pad material makes a difference as I mentioned earlier. The rear brakes are important too, and it may be a good idea to replace the linings to better material, but it won't improve the pedal feel. If the rears are adjusted properly, they'll take no more fluid to operate than new drums and linings do.