Google is not very clear on this, for reasons we may never know.
What you must do to start with is go to your Webmaster Tools Account and click on the messgaes link, found on the left side and look for something like this:-
If you have a Unnatural Inbound Links message like the one above, then you know 100% you have a Manual Penalty.
Other Manual penalties can be in relation to hidden links, inter linking your own sites, selling links and many more, but you should see a message warning you of such.
Warning: If you or someone else delete's that message or you move your WMT to another email address, then you may never know you have a manual penalty, this is a major floor in Webmaster Tools
Two major updates in the last two years have been the Panda & Penguin updates, both can result in Algorithmic penalties but you may never get a messsage to tell you there is a problem. One indication you have a problem could be an increase in errors showing up in various areas of Webmaster Tools.