The sweater needs restraint. A Hogwarts knit looks more like something from a secondhand college wardrobe when its house colors appear in small supporting details, rather than across an entire matching outfit. Less magic, better clothes. If you want the crest to read as a collegiate detail instead of a costume badge, give it ordinary clothes for company.
Corduroy does the work. Its ribbed surface brings another texture into the outfit, while brown, tobacco or olive trousers give the sweater a background that does not compete with its stripes or embroidery. Keep the fit relaxed. A straight leg with a modest break over the shoe suggests casual campus dressing, whereas trousers tucked into boots can push the same knit toward theatrical territory. Your accents should whisper. Try burgundy socks beneath a red-and-gold sweater, or a dark green belt with green-and-silver knitwear, choosing one echo rather than assembling every house color at once.
Loafers beat wizard accessories. Their familiar shape keeps the outfit grounded in everyday dressing, especially when you choose plain leather rather than footwear with buckles or decoration that competes with the sweater. Skip the matching scarf. A simple coat or an unstructured jacket can finish the outfit without adding another fictional reference, and an unbranded bag leaves the crest as the only obvious fandom signal. Actual age is optional. You are borrowing the visual ease of older collegiate clothing, not claiming that a newly bought sweater is an antique, so why dress as though you have a train to catch from Platform Nine and Three-Quarters?