Headcanon that when Chuuya first joined the mafia, he was a notorious clothes stealer.
It was a habit he learned from being with The Sheep. Hand-me-downs and clothes sharing was commonplace, especially in their earlier days. Chuuya's first outfit that wasn't rags was all shared clothes that Shirase had collected from The Sheep.
So when he joined the mafia, it was still second nature to him. Was the blazer he came home in the same one he left in? No, the cuff links aren't there, nevermind, it was still mafia clothes.
Albatross never had a problem with it, so Chuuya still didn't question it. Their clothes were similar sizes, swapping jackets and borrowing shirts was normal to them.
However as he got older, started developing his own style, and getting his clothes tailor made, he started caring more about losing them. He stopped taking other people's clothes realising it was not a common thing amongst his colleagues. Especially, after his promotion, it didn't feel right to take from his subordinates.
There was one person whose casual clothes still found their way into his wardrobe. It wasn't his fault if Dazai kept leaving sweaters and cardigans at Chuuya's apartment and never claiming them again. Honestly, sometimes Chuuya swore he only ever saw Dazai wear them once, to his place and then he left them completely forgotten about.
After Dazai defected, Chuuya kept wearing his clothes. Sometimes, on nights where he was in his feelings and perhaps a little bit tipsy, Chuuya would curl himself up inside one of Dazai's sweaters and fall asleep on his sofa.