I told my nginx server to listen only my own domains but it was listening to other domains as well. I added the following block to the top of my config file
before the original server settings:
server_name my.domain.com myother.domain.com;
This way any domain that is not in my list gets 404 by default. Hope it helps!