A Joint Fermi-GBM and LIGO/Virgo Analysis of Compact Binary Mergers from the First and Second Gravitational-wave Observing Runs