Is writing in time travel difficult?

I was thinking about some world settings in stories and I know when time travel is included in stories, it often comes off as something that seems very hard and complex to write about. You have to consider so many different angles, changes, ideas, growths happening with characters, and general changes within the world itself. I had considered including time travel in a story I had been sitting on for several years but I felt like this would be too complex for me to write about without ending up with huge plot wholes and underdeveloped setting scenarios. 

Do you think including time travel in a story is difficult? What can make it easier?