Can't join my buddy Minecraft world, Help?!

I'm trying to join my friends world in Minecraft but every time I try it, it tries to load but doesn't and then says connection failed (connection to server was lost. Exiting to the main menu.) Well I went back to my settings on my Xbox and tested the connection and it was fine, said I was connected and Its still saying connection failed. WTF!

Read more: Can't join my buddy Minecraft world, Help?!