How awful that it was your brother in law. Eww!
A creep started making those middle of the night calls to me when I was 14. I followed everyone’s “just hang up” advice for 5 years. It would stop for a while and I’d think I was rid of him, but it always started up again. Then when I was 19 he called again one night and I snapped. Instead of “just hanging up” I angrily told him that I knew who he was and where he lived (lies!) and that my boyfriend was going to hack his balls off. Creep never called again.
Ruining his masturbation fantasy was far more effective than “just hanging up.”