Introduction to Regression

Regression is a statistical and machine learning technique used to study the relationship between variables and to predict an output based on one or more input variables.

For example:

• Predicting salary based on years of experience
• Predicting house price based on its area and number of rooms
• Predicting whether a student will pass or fail based on attendance and marks

In regression terminology:

• Independent variable / Predictor / Input ($X$) → Variable used for prediction.
• Dependent variable / Response / Output ($Y$) → Variable to be predicted.

---

1. Linear Regression

Linear regression studies the relationship between one independent variable and a continuous dependent variable.

Example

Predicting a student's marks based on the number of hours studied.

$$\text{Hours Studied} \rightarrow \text{Marks}$$

The model tries to find the best-fitting straight line through the data.

The general form is:

$$Y = \beta_0 + \beta_1X$$

where:

• $\beta_0$ → Intercept
• $\beta_1$ → Slope
• $X$ → Input variable
• $Y$ → Predicted output

Example

$$\text{Marks} = 40 + 5(\text{Hours Studied})$$

---

2. Multiple Linear Regression

Multiple linear regression is used when we want to predict a continuous output using two or more independent variables.

Example

Predicting student marks using:

• Hours studied
• Attendance percentage
• Previous examination marks

The model can be represented as:

$$Y=\beta_0+\beta_1X_1+\beta_2X_2+\cdots+\beta_nX_n$$

Example

$$\text{Marks} = \beta_0+ \beta_1(\text{Study Hours})+ \beta_2(\text{Attendance})+ \beta_3(\text{Previous Marks})$$

Simple Linear Regression → One predictor
Multiple Linear Regression → Multiple predictors

---

3. Logistic Regression

Despite its name, logistic regression is mainly used for classification problems.

It predicts the probability that an observation belongs to a particular class.

Example

Predicting whether a student will:

• Pass (1)
• Fail (0)

based on variables such as attendance and study hours.

Unlike linear regression, the output is a probability between:

$$0 \text{ and } 1$$

For example:

$$P(\text{Pass})=0.85$$

This means that the model estimates an 85% probability that the student will pass.

A threshold, often 0.5, can then be used to convert the probability into a class prediction.

At threshold 0.50:

We compute a linear combination of inputs:

$$z={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_n{x}_n$$

Then apply the sigmoid function:

$\widehat{y}=\frac{1}{1+e^{-z}}$

• $\widehat{y}$= predicted probability
• Output is always between 0 and 1

Quick Comparison

Type	Number of Predictors	Output Type	Example
Linear Regression	One	Continuous	Predict marks from study hours
Multiple Linear Regression	Two or more	Continuous	Predict marks from hours, attendance, etc.
Logistic Regression	One or more	Categorical/Class	Predict pass or fail

Summary

Regression is used to understand relationships between variables and make predictions: linear regression predicts continuous values, multiple linear regression uses several predictors, and logistic regression predicts the probability of belonging to a class.