Introduction to Neural Networks

With this post, I welcome you to the world of neural networks where you are going to learn them in and out. If you are a deep learning rookie, this post will inspire you to get started in deep learning. Why are we witnessing this kind of build up for neural networks? Its because of their amazing applications. Some of their applications include image classification, face recognition, pattern recognition, automatic machine translation, and so on. So, let’s get started now.

Machine Learning is a field of computer science that provides computers the capability to learn and improve from experience without being programmed explicitly. Deep learning is a form of machine learning that uses a computing model that is highly inspired by the structure of the brain. Hence, we call this computing model as a Neural Network. A neural network is a computing system comprising highly interconnected and simple processing elements which process the information through their dynamic state response to external inputs. A ‘neuron’ is the fundamental processing element of a neural network. The neural network comprises a large number of neurons working simultaneously to solve specific problems.

This article explains the concept of neural networks and why they are a vital component in the process of deep learning. It also helps to let you know:-

  • The advantages of neural networks over conventional techniques
  • The architecture of Neural networks,
  • Training process of Neural Networks,
  • Network models and algorithms of Neural Networks

Also, the readers will come to know about the concept of deep neural networks.

Why Neural Networks Matter in Deep Learning?

Consider machine learning as a pack horse for processing information, then a carrot that draws the horse forward is the neural network. A system should not be programmed to execute a specific task for it to be able to learn truly; instead, it must be programmed to learn to execute the task. To accomplish this, the system uses deep learning(a more refined form of machine learning) which is based on neural networks. I have discussed deep learning in the previous section as well. With the help of neural networks, the system can perceive data patterns independently to learn how to execute a task.

Advantages of Neural Networks over Conventional Techniques

Depending on the strength of internal data patterns and the nature of the application, you can usually expect a network to train well. This is applied to problems where the relationships may be quite nonlinear or dynamic. Very often, the conventional techniques are limited by strict assumptions of variable independence, linearity, normality, etc. As neural network can capture various types of relationships, it enables the user to relatively easily and quickly model phenomena which otherwise may have been impossible or very difficult to explain.

Architecture of a Neural Network

Neural networks are modeled after the neuronal structure of the brain’s cerebral cortex but on smaller scales. They are usually organized in layers. Layers are comprised of many nodes which are interconnected and contain an activation function. The patterns are presented to the network through the input layer. This layer communicates to hidden layers(one or more in number) where the real processing is carried out through a system of weighted connections. Then, the hidden layers(neural hidden layer as shown in the below figure) are connected to an output layer(neural output layer as shown in the below figure) and it is the answer as depicted in the image shown below.

neural network architecture

Training in Neural Networks

Training a neuron involves applying a set of steps to adjust the thresholds and weights of its neurons. This kind of adjustment process(also known as learning algorithm) tunes the network so that the outputs of the network are very close to the desired values. The network is ready to be trained once it is structured for a specific application. The initial weights are selected randomly to begin this process. Then, the training or learning starts.

There are two approaches to training – unsupervised and supervised. In supervised training, the network is provided with the desired output in two ways. The first one involves manually grading the performance of the network and the second one is by allocating the desired outputs with the inputs. In unsupervised training, the network must make sense of the inputs without the help from outside. To put this in familiar terms, let’s consider an instance. Your kids are called supervised if you provide a solution to them during every situation in their life. They are called unsupervised if your kids make decisions on their own out of their understanding.

Most of the neural networks consist of some form of learning rule which alters the weights of connections according to the input patterns that are presented to it. Like their biological counterparts, the neural networks learn by example.

Neural networks use various kinds of learning rules. They are as follows.

  1. Hebbian Learning Rule – This learning rule determines, how to alter the weight of nodes of a network.
  2. Perceptron Learning Rule – The network begins its learning by allocating a random value to each weight.
  3. Delta Learning Rule – The modification in a node’s sympatric weight is equal to the multiplication of input and the error.
  4. Correlation Learning Rule – It is the supervised learning.
  5. Outstar Learning Rule – It can be used when it assumes that neurons or nodes in a network are arranged in a layer.

The Delta Learning Rule is often used by the most common class of neural networks known as BPNNs (backpropagation neural networks). Backpropagation implies the backward propagation of error.

Major Neural Network Models

The primary neural network models are as follows.

Multilayer perceptron – This neural network model maps the input data sets onto a set of appropriate outputs.

Radial Basis Function Network – This neural network uses radial basis functions as activation functions.

Both the above models are supervised learning networks, and they are used with one or more dependent variables at the output.

Kohonen Network – This is an unsupervised learning network. This is used for clustering process.

Neural Network Algorithms

As I stated earlier, the procedure used to perform the learning process in a neural network is known as the training algorithm. There are various training algorithms with different performance and characteristics. The major ones are Gradient Descent(used to find the function’s local minimum) and Evolutionary Algorithms(based on the concept of survival of the fittest or natural selection in biology).

Deep Neural Networks

Deep Neural Networks can be thought of as the components of broader applications of machine learning that involve algorithms for regression, classification, and reinforcement learning(a goal-oriented learning depending on interaction with the environment). These networks are distinguished from single-hidden-layer neural networks by their depth. This implies the number of node layers through which the data passes in a pattern recognition’s multi-step process.

Conventional machine learning depends on shallow networks that are composed of one output and one input layer with at most one hidden layer in-between. Including input and the output, more than three layers qualify as ‘deep’ learning. A deep neural network is shown in the below figure which has three hidden layers apart from the input and output layers. Hence, deep is a technical and strictly defined term that implies more than one hidden layer. Based on the previous layer’s output, each layer of nodes trains on a different feature set in deep neural networks.

layers in neural networks

Unlike most traditional machine learning algorithms, deep neural networks carry out automatic feature extraction without intervention. These networks can discover latent structures within unstructured(raw data), unlabeled data which is the majority of data in the world.

A deep neural network which is trained on labeled data can be applied to raw data. This gives the deep neural network access to much more input when compared with machine learning networks. This indicates higher performance as the accuracy of a network depends on how much data it is trained on. Training on more data results in higher accuracy.


So, we have come to the end of this post on neural networks. I have covered all the things that you should know to clearly understand the concept of neural networks. But, there is a lot more research that is going on in neural networks. Neural networks have broad applicability to business problems in the real world. They are currently used applied in various industries, and their applicability is getting increased day-by-day.

The primary neural network applications include stock exchange prediction, image compression, handwriting recognition, fingerprint recognition, feature extraction, and so on. Thus, neural networks are helping us solving complex real-world problems. In the coming years, you can see research that can break new ground in different domains through neural networks. I am sure that there will be quite a few more examples to highlight in the years to come. You can think of this as just the beginning of mainstream applications for deep learning.