The Wall Street Journal reported that the shorts are ridiculed by the partners of people who wear them, and they've even been banned at golf clubs.
"It seems that guys have had enough," says Marshal Cohen, a chief analyst with NPD Group, a market research firm. "With this movement towards the more athletic look, well, cargo shorts are the ones that are paying the price."
And over the past year, sales are down for the first time in a decade.
"When you're taking your significant other out to a dressier barbecue function, the last thing that they need to be doing is wearing a cargo short, right?" says Joseph Hancock, a design and merchandising professor at Drexel University.
What started out as a staple of military clothing became a trend that saw its peak in the late 1990s and early 2000s.
Hancock, who wrote his Ph.D. on the re-branding of cargo shorts, suggests their popularity can be boiled down to two things: "With the rise of gadgets, cargo shorts became really popular, because not only were they fashionable for the time period, they were functional."