Fallacies of Distributed Computing

 

1. The network is reliable;
2. Latency is zero;
3. Bandwidth is infinite;
4. The network is secure;
5. Topology doesn't change;
6. There is one administrator;
7. Transport cost is zero;
8. The network is homogeneous;

• Versioning is simple
• Compensating updates always work
• Observability is optional

Technology choices for Azure solutions

 

https://learn.microsoft.com/en-us/azure/architecture/guide/

Azure application architecture fundamentals

https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/technology-choices-overview

Technology choices for Azure solutions

When you design an Azure solution, you must select the right technologies for your workload across compute, data, networking, and other areas. The following resources include comparison matrices, flowcharts, and decision trees to help you evaluate your options and find the best match for your requirements.

Choose a compute service

The term compute refers to the hosting model for the computing resources that your application runs on. The following articles can help you choose the right technologies.

Article	Summary
Choose an Azure compute service	Decide which compute service best suits your application.
Choose an Azure compute option for microservices	Learn about two compute options for microservices: service orchestrator and serverless architecture.
Choose between traditional web apps and single-page applications (SPAs)	Learn how to choose between traditional web apps and SPAs.

Choose a container option

You can use multiple methods to build and deploy containerized applications in Azure. The following articles can help you evaluate container services.

Article	Summary
Choose an Azure container service	Evaluate which Azure container service best suits your specific workload scenarios and requirements.
Compare Azure Container Apps with other Azure container options	Learn when to use Container Apps and how it compares to other container options, including Azure Container Instances, Azure App Service, Azure Functions, and Azure Kubernetes Service (AKS).
Choose a Kubernetes at the edge compute option	Learn about trade-offs and considerations for various Kubernetes options for extending compute at the edge.

Choose a hybrid option

Many organizations need a hybrid approach for analytics, automation, and services because they host data both on-premises and in the cloud. The following articles can help you choose the best technologies for your scenario.

Article	Summary
Explore Azure hybrid options	Learn about Azure hybrid solutions, including alternatives to deploy and host hybrid services on-premises, at the edge, in Azure, and in other clouds.
Compare Azure Stack Hub to Azure	Learn the differences between Azure and Azure Stack Hub.
Compare Azure Local to Windows Server	Choose between Azure Local and Windows Server for your organization.
Choose drives for Azure Local and Windows Server clusters	Learn how to choose drives for Azure Local and Windows Server clusters to meet performance and capacity requirements.

Choose an identity service

Identity solutions help you protect your data and resources. The following articles can help you choose an Azure identity service.

Article	Summary
Compare Microsoft Entra ID and Domain Services identity solutions	Compare the identity services of Active Directory Domain Services, Microsoft Entra ID, and Microsoft Entra Domain Services.
Choose a hybrid identity authentication method	Choose an authentication method for a Microsoft Entra hybrid identity solution in a medium-sized to large organization.

Choose a storage service

Azure provides multiple storage services for different access patterns, performance tiers, and data types. The following articles can help you determine the best fit for your workload.

Article	Summary
Review your storage options	Review the storage options for Azure workloads.
Learn about Azure managed disk types	Learn about the disk types for Azure virtual machines (VMs), including Azure Ultra Disk Storage, Azure Premium SSD v2, Premium SSD, and Azure Standard SSD.
Choose a data transfer technology	Learn about Azure data transfer options like the Azure Import/Export service, Azure Data Box, Azure Data Factory, and command-line and graphical interface tools.

Choose a data store

Choose the right data store based on your data model, access patterns, scale, and consistency requirements. The following articles can help you choose a data solution.

Article	Summary
Evaluate data models	Learn how to evaluate Azure data store models based on workload patterns, scale, consistency, and governance.
Prepare to choose a data store	Learn how to choose the right Azure data store for your workloads by evaluating functional, performance, cost, and security requirements.
Choose a big data storage technology	Compare big data storage options in Azure. View key selection criteria and a capability matrix.
OLAP solutions	Learn about online analytical processing (OLAP) solutions for organizing large databases and supporting complex analysis without affecting transactional systems.
Learn about online transaction processing (OLTP) solutions	Learn about atomicity, consistency, and other features of OLTP, which manages transactional data and supports querying.
Learn about data lakes	Learn about data lake storage repositories, which can hold terabytes (TBs) or petabytes (PBs) of data in a native, raw format.
Choose a data pipeline orchestration technology	Choose an Azure data pipeline orchestration technology to automate pipeline orchestration, control flow, and move data.
Choose a search data store	Learn about the capabilities of search data stores in Azure and the key criteria for choosing one that best matches your needs.

Choose an analytics solution

Azure provides several services for analytics workloads, from data warehousing to real-time stream processing. The following articles can help you evaluate your options.

Article	Summary
Choose an analytical data store	Evaluate analytical data store options for big data in Azure.
Choose an analytical data store in Microsoft Fabric	Evaluate analytical data store options in Fabric based on data volumes and types, compute engine, ingestion, transformation, and query patterns.
Choose a data analytics and reporting technology	Evaluate big data analytics technology options for Azure.
Choose a batch processing technology	Compare technology choices for big data batch processing in Azure.
Choose a stream processing technology	Compare options for real-time message stream processing in Azure.

Choose an AI and machine learning service

Azure provides AI services, machine learning platforms, and model hosting options for many workloads. The following articles can help you select the right AI and machine learning technologies for your solution.

Article	Summary
Choose the right AI model for your workload	Learn strategies to help you select the best model for your AI workload, including key criteria and practical considerations for decision-making.
Choose an AI services technology	Learn about services that you can use in AI applications and data flows. Choose the right service for your use case.
Choose an Azure service for vector search	Learn how to decide which Azure service for vector search best suits your application.
Choose a natural language processing technology	Choose a natural language processing service for sentiment analysis, topic and language detection, key phrase extraction, and document categorization.
Compare Microsoft machine learning products and technologies	Compare options for building, deploying, and managing your machine learning models. Decide which Microsoft products to use for your solution.

Choose a networking service

Azure provides several networking services for traffic routing, load distribution, and virtual network connectivity. The following articles can help you evaluate your options.

Article	Summary
Compare load balancing options	Compare Azure load balancing services and select one to distribute traffic across multiple computing resources.
Compare virtual network connectivity options and spoke-to-spoke communication	Compare virtual network peering and VPN gateways for connecting Azure virtual networks. Learn implementation patterns for spoke-to-spoke communication in hub-and-spoke architectures.

Choose an integration and automation service

Azure provides several services for integrating systems and automating workflows, from code-first to designer-first approaches. The following article can help you evaluate your options.

Article	Summary
Compare Azure Functions, Azure Logic Apps, Power Automate, and WebJobs	Compare integration and automation services in Azure. Learn the differences between code-first and designer-first approaches for building workflows and orchestrations.

Choose a messaging service

Learn about the services that Azure provides to help you deliver events or messages throughout your solution.

Article	Summary
Compare messaging services	Learn about Azure Event Grid, Azure Event Hubs, and Azure Service Bus. Choose the best service for your scenario.
Learn about asynchronous messaging options	Learn about asynchronous messaging options in Azure, including the various types of messages and the entities that participate in a messaging infrastructure.

Choose an IoT option

Internet of Things (IoT) solutions use a combination of technologies to connect devices, events, and actions through cloud applications. The following articles describe IoT technology choices from Azure.

Article	Summary
Learn about Azure IoT services and technologies	Learn about Azure IoT services and technologies, including Azure IoT Hub, Azure IoT Operations, Azure IoT Edge, and Azure IoT Central. Choose between cloud-based and edge-based solution types.
Compare IoT Hub and Event Hubs	Review a comparison between Azure IoT Hub and Event Hubs that highlights functional differences and use cases. The comparison includes supported protocols, device management, monitoring, and file uploads.

Map services from other cloud providers to Azure

If you're migrating from another cloud provider or designing a multicloud solution, these articles map Amazon Web Services (AWS) and Google Cloud services to functionally similar Azure services.

Article	Summary
Azure for AWS professionals	Map AWS services to their Azure equivalents across category-specific articles that cover compute, storage, databases, networking, and more.
Azure for Google Cloud professionals	Look up Azure equivalents for Google Cloud services organized by technology category.

Linear Regression

Linear Regression is a mathematical method used to predict the value of a continuous target variable based on one or more input features.

Consider two related variables, x and y, where:

• x is the independent variable (feature/predictor).
• y is the dependent variable (target/output).

Given a dataset containing values of x and their corresponding values of y, linear regression attempts to predict unknown or future values of y for any given value of x.

Key Idea

Linear Regression assumes that the relationship between the input feature(s) and the output can be approximated using a straight line (or a hyperplane in higher dimensions).

---

Linear Regression Equation

For a single input feature, Linear Regression tries to find the best-fitting equation:

y = wx + b

where:

• w = Weight (Slope)
• b = Bias (Intercept)

This version is known as Simple Linear Regression.

When multiple input features are present, the equation becomes:

y = w₁x₁ + w₂x₂ + ... + wₙxₙ + b

This is known as Multiple Linear Regression.

Important

The learning process of Linear Regression consists of finding the values of the weights (w) and bias (b) that best fit the training data.

---

Hypothesis and Hypothesis Space

Every possible combination of the parameters w and b defines a different straight line.

Each such line (or equation) is called a Hypothesis.

The collection of all possible hypotheses is called the Hypothesis Space.

Think of it this way

Every possible straight line that could be drawn through the data belongs to the hypothesis space. The objective of training is to choose the single line that best represents the observed data.

The entire training problem therefore reduces to finding the best hypothesis.

---

Prediction and Error

Every hypothesis predicts one output value for every input value.

For each training sample:

Prediction = wx + b

The prediction is then compared with the actual value from the dataset.

The difference between the actual value and the predicted value is called the Error or Loss.

Loss = Actual Value − Predicted Value

Each training sample produces one loss value.

Since a dataset usually contains hundreds or thousands of observations, we need a single number that summarizes the prediction error across the entire dataset.

Definition

The overall error across the complete training dataset is called the Cost Function.

---

Common Cost Functions

Two cost functions are most commonly used in Linear Regression.

Loss Function	Description
L1 Loss (Mean Absolute Error)	Average of the absolute prediction errors.
L2 Loss (Mean Squared Error)	Average of the squared prediction errors.

1. Mean Absolute Error (L1 Loss)

The absolute value of every prediction error is computed before averaging.

MAE = Average(|Actual − Predicted|)

Since every error contributes proportionally, large errors are not excessively penalized.

2. Mean Squared Error (L2 Loss)

Each prediction error is squared before computing the average.

MSE = Average((Actual − Predicted)²)

Because the errors are squared, larger errors contribute much more heavily to the total loss than smaller ones.

---

Why L2 Loss Is Usually Preferred

Although both L1 and L2 losses are widely used, the L2 loss function offers two important mathematical advantages.

1. Differentiability

The L2 loss function is smooth everywhere. Its derivative exists at every point, including where the loss equals zero.

Why This Matters

Since the derivative exists everywhere, optimization algorithms such as Gradient Descent can easily determine the direction in which the parameters should be updated.

2. Convexity

The L2 loss function is also convex.

A convex function has only one global minimum and no local minima.

Important Clarification

L1 loss is also convex. The primary mathematical advantage of L2 over L1 is its smooth differentiability, not convexity itself.

Cost Functions

Each hypothesis produces a predicted value of y for every input value x.

The difference between the predicted value and the actual value is called the loss (or error).

The total loss over the complete training dataset is called the Cost Function.

Goal: Find the values of w and b that minimize the overall cost.

---

Common Loss Functions

1. Mean Absolute Error (L1 Loss)

The absolute value of each prediction error is calculated and then averaged.

MAE = |Actual − Predicted|

Characteristics:

• Easy to understand
• Robust against outliers
• Continuous everywhere
• Not differentiable at zero

Important:
L1 Loss is convex but contains a sharp corner at zero. Because of this kink, its derivative is undefined exactly at zero.

---

2. Mean Squared Error (L2 Loss)

Instead of taking the absolute value of the error, the error is squared. The squared errors are then averaged.

MSE = (Actual − Predicted)²

Why L2 Loss is Preferred

L2 loss has two extremely important mathematical properties.

1. Differentiability

L2 loss is smooth everywhere. Its derivative exists for every possible value, including zero.

Because the derivative exists everywhere, calculus-based optimization methods such as Gradient Descent work extremely well.

2. Convexity

The L2 cost function is convex.

A convex function contains only one minimum. Therefore there is no danger of getting trapped inside a local minimum.

Result:
There is a single global minimum that represents the best possible regression line.

---

Finding the Best Hypothesis

Since L2 loss is both differentiable and convex, the optimal values of w and b can be obtained in two ways.

Method 1 : Normal Equation

The derivative of the cost function with respect to w and b is computed.

The derivatives are set equal to zero and solved directly.

This analytical solution is called the Normal Equation.

---

Method 2 : Gradient Descent

Instead of solving the equations directly, Gradient Descent starts with random values of w and b and repeatedly improves them.

Each iteration moves the parameters in the direction that reduces the cost.

Gradient Descent is preferred for very large datasets because solving the Normal Equation becomes computationally expensive.

---

Why L1 Loss Requires Different Optimization

Since L1 loss is not differentiable at zero, the Normal Equation cannot be used.

Instead, optimization techniques such as:

• Subgradient Descent
• Linear Programming
• Simplex-based Optimization

are commonly used.

Notice that L1 loss is still continuous and convex. Its limitation is only the lack of differentiability at one point.

Key Assumptions of Linear Regression

Linear Regression performs well only when certain mathematical assumptions are reasonably satisfied.

Assumption	Description
Linearity	The relationship between the input variables and the target variable should be approximately linear.
Independence	Observations and their prediction errors should be independent of one another.
Homoscedasticity	The variance of the residual errors should remain approximately constant across all values of the independent variables.
Normality	Residual errors should be approximately normally distributed.

Note

Linear Regression is surprisingly robust. Minor violations of these assumptions often do not significantly affect prediction accuracy, especially when working with large datasets.

---

Evaluating a Linear Regression Model

After training a regression model, we need to evaluate how well it fits the training or test data. Several evaluation metrics are commonly used.

Metric	Purpose
R² (Coefficient of Determination)	Measures how much of the variation in the target variable is explained by the model.
Mean Absolute Error (MAE)	Average absolute prediction error.
Root Mean Squared Error (RMSE)	Square root of the Mean Squared Error. Larger errors receive greater penalty.

Rule of Thumb

• Higher R² is generally better.
• Lower MAE is better.
• Lower RMSE is better.

---

Overfitting and Regularization

When the number of input features becomes very large, a regression model may begin to memorize the training data instead of learning the underlying relationship.

This phenomenon is called Overfitting.

An overfitted model usually performs extremely well on the training data but performs poorly on previously unseen test data.

To reduce overfitting, additional penalty terms can be added to the cost function. This technique is known as Regularization.

---

Common Regularization Techniques

Algorithm	Penalty Added	Characteristics
Ridge Regression	L2 Penalty	Shrinks coefficient values but rarely makes them exactly zero.
Lasso Regression	L1 Penalty	Can reduce some coefficients completely to zero, effectively performing feature selection.
ElasticNet	L1 + L2	Combines the advantages of both Ridge and Lasso.

Key Point

Regularization intentionally allows a small increase in training error in exchange for much better performance on unseen data.

---

scikit-learn and pytorch implementation

The final section explains the default behavior of Linear Regression in two popular Machine Learning frameworks:

• scikit-learn
• PyTorch

It also discusses how regularization is enabled in each framework and the default settings used by Ridge, Lasso, and ElasticNet.

---

Default Behavior in scikit-learn

The LinearRegression class in scikit-learn implements Ordinary Least Squares (OLS).

By default, it applies no regularization whatsoever.

Default Behavior

• LinearRegression() → Ordinary Least Squares (No Regularization)

If regularization is required, scikit-learn provides separate algorithms.

Model	Regularization	Default Parameters
LinearRegression()	None	Ordinary Least Squares
Ridge()	L2 (Ridge)	alpha = 1.0
Lasso()	L1 (Lasso)	alpha = 1.0
ElasticNet()	L1 + L2	alpha = 1.0 l1_ratio = 0.5

Important

scikit-learn never applies regularization automatically. You must explicitly choose Ridge, Lasso, or ElasticNet if regularization is desired.

---

Default Behavior in PyTorch

PyTorch follows a fundamentally different philosophy.

Unlike scikit-learn, PyTorch does not provide a dedicated "Linear Regression" algorithm. Instead, users construct the model themselves using neural network building blocks.

Typical components include:

• nn.Linear for the model
• nn.MSELoss() for the loss function
• An optimizer such as SGD or Adam

Because every component is built explicitly by the developer, PyTorch applies no regularization by default.

---

Adding Regularization in PyTorch

Regularization is typically introduced in one of two ways.

Method 1 : Add the Penalty to the Loss Function

The loss function can be manually extended by adding either an L1 or L2 penalty.

loss = mse_loss + λ × Σ(weights²)

The above expression corresponds to Ridge (L2) regularization.

Similarly, Lasso (L1) regularization can be implemented by summing the absolute values of the weights.

---

Method 2 : Use weight_decay

Most PyTorch optimizers expose a parameter called weight_decay.

This parameter automatically performs L2 (Ridge) regularization during optimization.

optimizer = torch.optim.Adam( model.parameters(), lr=0.001, weight_decay=0.0001 )

The default value is weight_decay = 0 which means regularization is disabled by default.

---

Does PyTorch Support L1 Regularization?

Unlike L2 regularization, PyTorch optimizers do not provide a built-in parameter for L1 regularization.

If L1 (Lasso-style) regularization is required, it must be added manually to the loss function.

Summary

• L2 → Built into optimizers via weight_decay.
• L1 → Must always be added manually.

---

Framework Comparison

Framework	Default Regularization	How to Enable
scikit-learn	None	Use Ridge(), Lasso(), or ElasticNet().
PyTorch	None	Use weight_decay (L2) or manually modify the loss function (L1).

---

To Summarize

• Linear Regression finds the best-fitting line through the training data.
• The quality of a hypothesis is measured using a Cost Function.
• L2 (Mean Squared Error) is the most commonly used loss because it is both smooth (differentiable) and convex.
• The optimal parameters can be obtained analytically using the Normal Equation or iteratively using Gradient Descent.
• Regularization reduces overfitting by penalizing large model weights.
• Ridge uses an L2 penalty, while Lasso uses an L1 penalty.
• scikit-learn and PyTorch both disable regularization by default; it must be explicitly enabled by the developer.