What is Transfer Learning?

During transfer learning, the knowledge leveraged and rapid progress from a source task is used to improve the learning and development to a new target task. Read on for a deeper dive on the subject.

Transfer Learning is a machine learning method where the application of knowledge obtained from a model used in one task, can be reused as a foundation point for another task.

Machine learning algorithms use historical data as their input to make predictions and produce new output values. They are typically designed to conduct isolated tasks. A source task is a task from which knowledge is transferred to a target task. A target task is when improved learning occurs due to the transfer of knowledge from a source task. 

During transfer learning, the knowledge leveraged and rapid progress from a source task is used to improve the learning and development to a new target task. The application of knowledge is using the source task’s attributes and characteristics, which will be applied and mapped onto the target task. 

However, if the transfer method results in a decrease in the performance of the new target task, it is called a negative transfer. One of the major challenges when working with transfer learning methods is being able to provide and ensure the positive transfer between related tasks, whilst avoiding the negative transfer between less related tasks. 

Before we dive into the methodology behind transfer learning, it is good to know the different forms of transfer learning. We will go through three different types of transfer learning scenarios, based on relationships between the source task and target task. Below is an overview of the different types of transfer learning:

  Inductive Transfer Learning: In this type of transfer learning, the source and target task are the same, however, they are still different from one another. The model will use inductive biases from the source task to help improve the performance of the target task. The source task may or may not contain labeled data, further leading onto the model using multitask learning and self-taught learning. 

Unsupervised Transfer Learning: I assume you know what unsupervised learning is, however, if you don’t, it is when an algorithm is subjected to being able to identify patterns in datasets that have not been labeled or classified. In this case, the source and target are similar, however, the task is different, where both data is unlabelled in both source and target. Techniques such as dimensionality reduction and clustering are well known in unsupervised learning. 

Transductive Transfer Learning: In this last type of transfer learning, the source and target tasks share similarities, however, the domains are different. The source domain contains a lot of labeled data, whereas there is an absence of labeled data in the target domain, further leading onto the model using domain adaptation. 

  Fine-tuning is an optional step in transfer learning and is primarily incorporated to improve the performance of the model. The difference between Transfer learning and Fine-tuning is all in the name.

Transfer learning is built on adopting features learned from one task and “transferring” the leveraged knowledge onto a new task. Transfer learning is usually used on tasks where the dataset is too small, to train a full-scale model from scratch. Fine-tuning is built on making “fine” adjustments to a process in order to obtain the desired output to further improve performance.