Opening Chat

Do you often hear friends say: "I want to learn machine learning, but don't know where to start"? As a Python programmer, I deeply relate to this. I remember when I first encountered machine learning, I was completely lost. Later I discovered that rather than diving into complex deep learning models right away, it's better to start with the basics like linear regression to build a solid foundation.

Today, I want to share with you how to implement a linear regression model from scratch using Python. Why choose linear regression? Because it's one of the most fundamental and important algorithms in machine learning. Understanding the principles of linear regression will make learning other machine learning algorithms twice as effective.

Theoretical Foundation

Before we start coding, we need to understand some basic concepts. What is the essence of linear regression? Simply put, it's about finding a straight line that best fits our data points. This line can be represented by a simple equation: y = wx + b, where w is the slope and b is the intercept.

You might ask, how do we determine if this line is the "best"? This is where an important concept comes in: Mean Squared Error (MSE). MSE is the average of the squared differences between predicted and actual values. The mathematical formula is: MSE = (1/n) * Σ(y_predicted - y_actual)². Our goal is to minimize this MSE value.

Code Implementation

Let's implement this model step by step. First, we need some basic tools:

import numpy as np
import matplotlib.pyplot as plt


np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

Here I used NumPy to generate 100 random data points. See, I set the random seed to 42, so we get the same random data every time we run the code, making it easy to reproduce results.

Here's the core part of the model:

class LinearRegression:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.weights = None
        self.bias = None
        self.history = {'loss': []}

    def fit(self, X, y):
        # Initialize parameters
        n_samples, n_features = X.shape
        self.weights = np.zeros((n_features, 1))
        self.bias = 0

        # Gradient descent
        for _ in range(self.n_iterations):
            # Forward propagation
            y_predicted = np.dot(X, self.weights) + self.bias

            # Calculate gradients
            dw = (1/n_samples) * np.dot(X.T, (y_predicted - y))
            db = (1/n_samples) * np.sum(y_predicted - y)

            # Update parameters
            self.weights -= self.learning_rate * dw
            self.bias -= self.learning_rate * db

            # Record loss
            loss = np.mean((y_predicted - y) ** 2)
            self.history['loss'].append(loss)

    def predict(self, X):
        return np.dot(X, self.weights) + self.bias

This code might look complicated, let's analyze it:

1. In the init method, we define two hyperparameters: learning rate and number of iterations. The learning rate determines the step size of each parameter update, and the number of iterations determines how many rounds the model trains.

2. The fit method is the core of model training. We use gradient descent algorithm to find optimal parameters. In each iteration, we:

3. Calculate predicted values with current parameters
4. Calculate gradients of the loss function with respect to parameters
5. Update parameters based on gradients

6. Record current loss value

7. The predict method is much simpler, just making predictions using trained parameters.

Model Testing

Let's see how our model performs:

model = LinearRegression(learning_rate=0.01, n_iterations=1000)
model.fit(X, y)


plt.scatter(X, y, color='blue', label='Data points')
plt.plot(X, model.predict(X), color='red', label='Prediction')
plt.xlabel('X')
plt.ylabel('y')
plt.title('Linear Regression Result')
plt.legend()
plt.show()


plt.plot(model.history['loss'])
plt.xlabel('Iteration')
plt.ylabel('Loss')
plt.title('Training Loss Over Time')
plt.show()

Practical Insights

I have several insights from implementing this model that I'd like to share:

1. Data standardization is important. Although we didn't standardize data in this simple example, in real projects, standardization can improve model training effectiveness and convergence speed.

2. Learning rate selection is crucial. If the learning rate is too large, the model might not converge; if too small, training will be too slow. I suggest trying different learning rates to see how the model performs.

3. Pay attention to the loss curve. If loss keeps decreasing, it means the model is continuously improving; if loss fluctuates or increases, you might need to adjust the learning rate.

Future Outlook

Linear regression is just the tip of the iceberg in machine learning. After mastering linear regression, you can continue learning more complex algorithms like logistic regression, decision trees, random forests, etc. Each algorithm has its characteristics and applicable scenarios, just like we programmers need various tools in our toolbox.

Have you wondered why many companies are hiring machine learning engineers now? Because machine learning is changing how we live and work. From recommendation systems to autonomous driving, from image recognition to natural language processing, machine learning applications are everywhere.

As Python programmers, we have a natural advantage. Python's rich machine learning ecosystem, like NumPy, Pandas, Scikit-learn, etc., provides us with powerful tools. I suggest learning to use these tools while mastering the basic principles.

What other practical scenarios do you think linear regression models could be used in? Feel free to share your thoughts in the comments.