It's tough being a landlord and many people don't see that. As you say they just think you are rich.
We had 2 rental properties in the UK, both of which never really made much more than the mortgage repayments. Both times we bought these properties at the wrong times. When we sold a few years ago, we sold for about the same as we paid for them. We were just glad to be rid of them and the stress.
We are now hoping our french property investment will be a better one. Buying in the Alps a popular holiday resort means we are now entering the AirBnB market instead. It will be interesting to see where this takes us.