Ah, thank you for the explanation. That makes sense why it would start sometimes and other times it would not . You could actually hear the fuse go when it happened.
Your starter solenoid current is too low, try increasing it a bit by replacing the wire from the relay to the solenoid with something a bit larger, if that doesn't do try cleaning the ignition switch contacts.
If all else fails add a new fuse to the battery and feed both start relays Yellow wire, this will add 25% more current and make it very reliable.
The solenoid would take 50 Amps if you gave it a good supply, you probably have 25 or Less.