How about looking at it this way Chris, how would you know you were causing them to stumble unless they told you. Those of the world tend to revel in worldlinesses, and would not come to you and ask you to help them build their Spiritual strength as, apparently those offended by the eating of sacrificed meat did. If someone has a problem, they should state it to you. beyond that, you have no responsibility for the thoughts of others. There are popular misconceptions about people who drive sports cars, teenagers with cell phones, people with southern accents, etc,etc. Whatever the misconception, that doesn't make it true. Your only responsibility is the modesty of your own conduct.
Of relevance on this topic: the wide wear of robes & tunics throughout the Roman empire, and I don't seem to remember any Biblical reference to that, nudity at Olympic or other games like in war. The only close thing I can thing I can think of is the appeal for men not to wear long hair; which is better translated as "effeminate." Long was actually also customary.
How about a "legal" P.C comparison we've probably all been preached at about. Men shouldn't sexually harass women. If the woman never tells the man she feels sexually harassed, how can he be "convicted" of it? So two points: either he didn't mean to and then knows what it is she feels offensive and can then refrain, or he was doing it deliberately and better stop. So, from the Biblical admonition about meat, if no one (or through you pastor, etc) tells you it's a stumbling block for them; then the thoughts in the privacy of their own mind, are their own. Enough women have certainly been rebuked for their (lack of) modest clothing, why shouldn’t equal treatment be equal for all?