Majo to Neko no Kakurezato

"Catzauberer"
A hidden village of witches protected by magical barrier where
Catgirls and Witches live in secret.

One night, the protagonist followed his pet cat into a narrow alley and
accidentally stumbled upon a barrier opening, leading him into a village.

The protagonist was rescued by a Catgirl and ended up living in this village with
only women until he found his pet cat.