WHY is your idle speed so high? Seems totally unreasonable.
Seems like a booster problem.
Again, seems like a booster problem. No vacuum reserve. One pedal-push exhausts the vacuum supply.
The lack of vacuum reserve causes the engine to lean-drop when the brake pedal is pushed, leading to driveability issues.
Unsafe? No, not really. Just that you'd have no power assist on the brakes, so you'd need a heap of "foot-pounds" on the pedal to get the car to stop.
The diaphragm isn't damaged with lack of vacuum. Although, you may have that--or other--faults in the booster already.
Maybe I'm wrong. I don't know dickety-doo about Holley injection systems. But to me, this is a mechanical problem, not a "computer tuning" problem.