Gradient Descent Gradient Descent is the most basic but most used optimization algorithm. It’s used heavily in linear regression and classification algorithms. Backpropagation in neural networks also uses a gradient descent algorithm.

Introduction: In optimization of a design, the design objective could be simply to minimize the cost of production or to maximize the efficiency of production. An optimization algorithm is a procedure which is executed iteratively by comparing various solutions till an optimum or a satisfactory solution is found.

How to choose the right optimization algorithm?

1. Minimize a function using the downhill simplex algorithm.
2. Minimize a function using the BFGS algorithm.
3. Minimize a function with nonlinear conjugate gradient algorithm.
4. Minimize the function f using the Newton-CG method.
5. Minimize a function using modified Powell’s method.

Specifically, you learned: Adam is a replacement optimization algorithm for stochastic gradient descent for training deep learning models. Adam combines the best properties of the AdaGrad and RMSProp algorithms to provide an optimization algorithm that can handle sparse gradients on noisy problems.

An optimization model is a translation of the key characteristics of the business problem you are trying to solve. The model consists of three elements: the objective function, decision variables and business constraints.

Adam is the best optimizers. If one wants to train the neural network in less time and more efficiently than Adam is the optimizer. For sparse data use the optimizers with dynamic learning rate.

An Example : Whale optimization algorithm 1- Encircling prey 2- Bubble-net attacking method (exploitation phase) 3- Search for prey (exploration phase) Behavior of Whale 13. 13 Workshop on Intelligent System and Applications (ISA’17), Faculty of Computers and Informatics, Benha University.

Optmization techniques 1 OPTIMIZATION TECHNIQUES 2 Definition:  An optimization is the act of achieving the best possible result under given circumstances. 3 3 optimization Reduce the cost Safety & reduce the error reproducibilit y Save the time Why Optimization is necessary?

The formulation of an optimization problem begins with identifying the underlying design variables, which are primarily varied during the optimization process. A design problem usually involves many design parameters, of which some are highly sensitive to the proper working of the design.

Meta-heuristic Algorithms Meta-heuristic is a general algorithmic framework which can be applied to different optimization problems with relatively few modifications to make them adapted to a specific problem. 11. 11 Workshop on Intelligent System and Applications (ISA’17), Faculty of Computers and Informatics, Benha University.