The starting point in Machine Learning is the understanding of the different types of learning algorithms. These were inspired by the way human mind tries to learn and make sense out of the various observations. These are perhaps the "Newton's laws" of Machine Learning. We must understand them thoroughly before proceeding further.
Based on the problem, available data, and the computational power at hand, there are different types of machine learning algorithms that we use
Overfitting is one of the major pitfalls we have to tackle when working on any machine learning algorithm. Researchers have studied this problem pretty well and have provided us some useful best practices to keep it at bay.
There are several important algorithms that help us with Supervised and Unsupervised Learning. And these algorithms continue to evolve as the research continues. There are no good or bad algorithms. Each has its own intricacy that makes it good for a particular scenario. One has to understand the given scenario and then use the appropriate algorithm.
For an efficient training process, it is improtant that we use only what is required. Dimensionality reduction is an important step in ensuring that.
As one would expecte, the Non-Linear techniques are more effective and expensive compared to the linear techniques.
Thanks to the extensive research and contribution of the Open Source community, most machine learning algorithms have a ready made implementation available in a choice of languages. That does not require any more effort. But that does not make machine learning any simpler. The main job is choosing the right algorithm for the data available. For doing that, we need to understand the algorithms in depth. We need to understand the pitfalls so that we can work around them.