There are two main modified server types: Bukkit and Forge. The first can only load plugins, whereas the other works with mods. If you are using a modpack, then your server is based off Forge.
When using Bukkit (or its cooler brothers, Spigot and Paper), your server should be able to load plugins out-of-the-box. If for some reason your plugins aren't working, chances are you installed a version not compatible with the server version. Either that, or they're not properly configured.
However, if you have a Forge server, things might be a bit more complicated, but you'd still have a few solutions:
- You can use mods instead of plugins. In fact, many popular plugins, like Essentials, may have a mod version for Forge servers (e.g. Forge Essentials).
- You can install SpongeForge, a mod that will let your server support the following plugins: check them out. Please note that Sponge plugins must be installed into the mods directory.
- You can install Thermos, a modified version of Forge that will enable support for Bukkit plugins. Unfortunately, Thermos only works with older servers (1.7.10), so if you are using a relatively recent modpack, this method is not what you are looking for.