Mathematical programming tends to model a problem directly as a mathematical formulation. Namely, a mathematical formulation consists of an objective to be minimized (or maximized), and a set of equalities/inequalities that confine the domain of possible solutions for the problem. The formulation needs to give a precise description of our exact task. Solving the problem is identical to solving the formulation.
In contrast with mathematical programming, employing a machine learning approach for problem solving usually does not yield a model precisely portraying the problem. Instead, the modeling approach in the machine learning context refers to choosing an appropriate representation. The representation can be interpreted as a set of models and the set is called hypothesis space. If our problem itself is outside the hypothesis space then the learning approach wouldn’t work such that one needs to switch to another representation.
Because machine learning does not model the problem itself exactly, the goal that we target inside our machine learning approach may differ from that we indeed want to evaluate in our problem. While in mathematical programming, we optimize exactly the metric that we would like to improve in our problem.
Both machine learning and mathematical programming need optimization. For the latter, optimization is used to solve the formulation. For the former, it is used to search among the hypothesis space to identify the most suitable model. (Remark that the most suitable model does not necessarily mean the evaluation function needs to be solved to optimality.)
(to be continued)