Everybody can remember outfits like Celine Dion's backwards Dior tuxedo... these 'worst' dresses become part of pop culture, and we keep referencing them.
If you keep everything in the same palette of tone-on-tone neutrals, even a laid-back outfit of a slouchy sweater, jeans, and sneakers gives you a longer, leaner look.