Anime is a very broad genre, aiming at small kids, teenager to young adults; and since some decades more and more also older adults. Ghibli for example is more family-orientated than just children. So it highly depends on which anime you enjoy and for what reasons, and what you make of it. Comparable here are Harry Potter-, Marvel- or Star Wars-Movies. Nobody would say liking these makes you unattractive, but most people would also not say it makes you attractive (fans of those franchises/genre are the obvious exception).

The actual thing is, you probably got it the wrong way. Not anime makes you unattractive, the lack of anything attractive makes you unattractive.