ReLU Activation Function

 ReLU stands for Rectified Linear Unit. ReLU activation function is one of the most used activation functions in the deep learning models. ReLU function is used in almost all the convolutional neural networks or deep learning models.

The ReLU (Rectified Linear Unit) function is an activation function that is currently more popular as compared with the sigmoid function and the tanh function.



Advantages of tanh function

When the input is OK, no gradient saturation problem.

The calculation speed is very quickly. The ReLU function has only a direct relationship. Even so forward or backward, much faster than tanh and sigmoid.(tanh and Sigmoid  you need to calculate the object, which will move slowly.)

Disadvantages of tanh function

When the input is negative, ReLU is not fully functional, which means when it comes to the wrong number installed, ReLU will die. This problem is also known as the Dead Neurons problem. While you are forward propagation process, not a problem. Some areas are sensitive while others are present unsympathetic. But in the back propagation process, if you enter something negative number, the gradient will be completely zero, with the same problem as sigmoid function and tanh function.

We find that the result of ReLU function can be 0 or positive number, which means that ReLU activity is not 0-centric activity.

ReLU function can only be used within Hidden layers of a Neural Network Model.

To overcome the Dead Neurons problem of ReLU function another modification was introduced which is called Leaky ReLU. It introduces a small slope to keep the updates alive and overcome the dead neurons problem of ReLU.

Another variant was made from both ReLu and Leaky ReLu called which is known as Maxout function which we will be discussing in details in other articles.


Comments

Post a Comment

Popular posts from this blog

Python - Queue

 We generally remain in line where we need to sit tight for administration. The equivalent goes for the line information structure where the things are masterminded in the line. line varieties are controlled by how the thing is added or taken out. Things just permitted in one end and eliminate from the opposite end. It's the earliest in, earliest out strategy. The line can be utilized utilizing a python list, were input () and pop () strategies are utilized to add and eliminate things. Since information things are constantly added toward the finish of the line.  Tasks Of a Queue  1) Enqueue: Adds a component to the furthest limit of the line. Flood condition - Queue is full  2) Dequeue: Removes a component from the front of the line. Sub-current condition - Queue is vacant  3) Front: Get first thing  4) Rear: Get last thing  Execution  We can execute line utilizing information design and python modules as follows:  1 ) list  2) collectio...
Read more

Namespaces and Scope in Python

In this article, we will think about the namespaces in python , kinds of namespaces, and degrees in python. In python programming, everything is viewed as an item. Name is only the identifier of an item. Also, space is a location in the primary memory related with that article. We can characterize namespace as an assortment of names related with the location in the principle memory. There are three kinds of namespaces, Built-in Namespace, Global Namespace, and Local Namespace. Each lower namespace can get to the upper namespaces. Also, the extent of factors in python relies upon the namespaces.  Namespace in Python  Each substance in python is viewed as an item. We give a few names to each protest like variable, class, and capacity for recognizable proof. Frequently these names are known as identifiers. Thus, the name is only the identifier. This load of names and where we use esteem are put away in the primary memory at a one of a kind area. This area is known as space. So th...
Read more

Python Identifiers

Image
  We will see the rules to define identifiers, and all the best practices to follow while defining Python identifiers. Let’s start with the definition of identifiers. What is Python Identifier? “An identifier is a name given to an entity”. In very simple words, an identifier is a user-defined name to represent the basic building blocks of Python. It can be a variable, a function, a class, a module, or any other object. Naming Rules for Identifiers Now you know what exactly identifiers are. So, how do we use them? We can’t use anything, there are some certain rules to keep in mind that we must follow while naming identifiers. 1. The Python identifier is made with a combination of lowercase or uppercase letters, digits or an underscore. These are the valid characters. Lowercase letters (a to z) Uppercase letters (A to Z) Digits (0 to 9) Underscore (_) or square brackets (*), followed by single quotes as text: "%u", "%m" = "-32,-192". These two examples tell...
Read more