How to make modded,plugin server for MINECRAFT?

I couldn't quite find another similar question that fit exactly the way I liked so I figured I would just have to ask myself. We used to do vanilla, but have since gotten bored, so please don't bring up just doing vanilla unless you want me to lose it.
So my brother and I are looking to make our own personal modded/plugin server for a group of friends. The PC running the server is 64 bit windows 10, we have 32gb of ddr4 ram, and more than enough processing power for anything else needed, so for the most part that bit should all be taken care of.
The problem is, is that in past we have tried doing this and there are always some sort of errors and then of course the crashing starts. I wasn't the one doing it, my brother was, so now that I'm taking over I want to make sure I do it right and we don't end up putting tons of hours in just to lose data somehow.
What I'm wondering is how to make a server that allows both plugins and mods. What should I used? Bukkit, Spigot, Forge, cauldron, mod something api, anything else? What is and what isn't compatible on what I just listed? Does the version of which I use matter? Please give me your recommendations based off of what you would want a person who is just learning how to do servers, not simply based off of opinion.

Read more: How to make modded,plugin server for MINECRAFT?