Blob core windows net

In November 2023, I presented at CHCON «Go by Example: Creating a C2 framework while trolling Microsoft». The recording is available on YouTube if you’d prefer to watch. I’ve written the core points of that presentation into this post.

Background

If you’ve ever talked to me about smart homes and automation or read some of my other blog posts, you’ll have seen that I have a fairly complex internal network architecture.

(I have waaaayyyyyy too many computers, and no time to manage them all.)

Which is why in early 2023, I set myself a «simple» task: I want a unified management portal for my fleet of IoT, so that I don’t have to spend all my free time running around going sudo apt update && sudo apt upgrade -y into everything.

Specifically, I wanted:

• A single agent process that I could load onto all my devices
• A simple microservice backend to coordinate running tasks on the agents
• A simple cli tool that I could use to query and run commands at scale

I could have used any of the countless RMM tools available to do this, I had an idea of how to do it differently. Because, as the title of the talk implies, I had a bit of Microsoft trolling to do 😉.

The Design

In my day job I write a lot of Python, and am quite familiar with using Azure Functions, so my first thought was to simply stick to what I know.

But I’ve recently started reimplementing parts of my home automation stack using Go. Mostly because I enjoy learning new things, and while I’d played around with contributions to other Go projects, I wanted to understand a fully-worked Go example. But also because Go compiles into a neat fully-packaged binary, which is ideal for my use-cases, where I have task-specific software that runs on home automation devices; in addition to having lots of apt updates, I was also tired of python3 -m venv .env && source .env/bin/activate && pip install --upgrade pip && pip install --upgrade -r requirements.build.txt && pip freeze > requirements.txt.

And while I could have used Go to write both the agent and the backend microservice, I decided that the future is truely code-less architectures, and decided to build the server side using Azure Storage Blobs.

Which… isn’t as crazy as you might first think. If you start by understanding the API design, the agent only needs a couple basic APIs:

• a PUT to report it’s current status
• a GET to fetch instructions (like if it needs to run an update command)
• a PUT to upload results (e.g., if I want to fetch a file remotely)
• a POST to capture logs for diagnostics

All of these can be natively run as storage GET/PUT/POST operations. I can create a single Blob storage Container to hold all the agents, and use SAS tokens to segment each agent, so that a malicious agent can’t interfere with others.

Картинка с сайта: www.jlaundry.nz

Using a Storage account as an API backend has numerous benefits, including:

• No-code, true server-less; nothing on the backend requires any maintenance whatsoever
• Cloud-native scalability, I can scale from 1 to 1bn devices instantly
• Cost effective, at ~$0.000004 (NZD) per operation!
• Super fast, no waiting for stopped applications to start
• No firewall rules to add

If you’d like to see it in action, go watch the YouTube video. No sorry, I haven’t open-sourced the code. 🙂

No firewall rules to add?

No firewall rules to add, because you’ve already added them.

Remember that Microsoft trolling I promised?

See, Azure Blob Storage is so popular with Microsoft product teams, that just about every Microsoft product asks you to add *.blob.core.windows.net. Seriously, go look at the Microsoft documentation. Some highlights include:

• Office 365
• Windows Delivery Optimisation
• Entra Connect
• Defender for Endpoint, although at least they say: «Don’t exclude the URL *.blob.core.windows.net from any kind of network inspection»
• Defender for Cloud Apps
• Visual Studio
• PowerApps
• Azure Backup
• Azure Site Recovery

Does this count as a vulnerability?

I’ve tried to raise this with Microsoft, including people I know who work there, and have always gotten a lukewarm response.

This is not exactly a new technique. There are documented examples of attackers using Azure services to host phishing sites. I even managed to find an example of someone who tried to report abuse to Microsoft, but gave up, and posted in the Learn forums in frustration. But as far as I can find, I’m the first to use Azure Storage as a RMM / C2.

The point is that each Azure Storage account should be treated like any other website, with all the typical controls that you would expect. Everyone should expect services that are available to anyone to be abused. In particularly risk-adverse organisations, this means accounts (domains) should be checked to see if they’re newly seen, or are hosting content that contains brand impersonations.

But while this is a known issue, Microsoft product teams either aren’t aware, or aren’t taking steps to help reduce the impact Instructing IT teams to explicitly trust every Azure Storage account, by adding *.blob.core.windows.net to security bypass lists, is downright dangerous. Especially in situations where you’re trying to create a strict outbound control list, like when your company is being hit by ransomware, and you’re trying to cut off internet access while still allowing core tools (like EDR, and directory sync) to function.

In my opinion, Microsoft need to do a stocktake of their official Azure Storage accounts, and at least put them behind a CNAME of a Microsoft-only domain like *.blob.cloud.microsoft. Storage Accounts support custom domains, and it’s not even very difficult to do:

Картинка с сайта: www.jlaundry.nz

Epilogue: how many Microsoft official Azure Storage accounts are there?

Spoiler: I don’t know.

I’ve been trying to find out though, and through observing network traffic I’ve found that there are at least 209 Storage accounts used for OneDrive for Business, 34 Storage accounts used for Defender for Endpoint.

Azure Storage Series

1. Azure Storage – Step 1: Create a storage account
2. Azure Storage – Step 2: Manage your storage account
3. Azure Storage – Step 3: Get started with Azure Blob storage using .NET

Overview

Azure Blob storage is a service that stores unstructured data in the cloud as objects/blobs. Blob storage can store any type of text or binary data, such as a document, media file, or application installer. Blob storage is also referred to as object storage.

What is Blob Storage?

Azure Blob storage is a service for storing large amounts of unstructured object data, such as text or binary data, that can be accessed from anywhere in the world via HTTP or HTTPS. You can use Blob storage to expose data publicly to the world, or to store application data privately.

Common uses of Blob storage include:

• Serving images or documents directly to a browser
• Storing files for distributed access
• Streaming video and audio
• Storing data for backup and restore, disaster recovery, and archiving
• Storing data for analysis by an on-premises or Azure-hosted service

Blob service concepts

The Blob service contains the following components:

• Storage Account: All access to Azure Storage is done through a storage account. This storage account can be a General-purpose storage account or a Blob storage account which is specialized for storing objects/blobs.
• Container: A container provides a grouping of a set of blobs. All blobs must be in a container. An account can contain an unlimited number of containers. A container can store an unlimited number of blobs. Note that the container name must be lowercase.
• Blob: A file of any type and size. Azure Storage offers three types of blobs: block blobs, page blobs, and append blobs.Block blobs are ideal for storing text or binary files, such as documents and media files. Append blobs are similar to block blobs in that they are made up of blocks, but they are optimized for append operations, so they are useful for logging scenarios. A single block blob or append blob can contain up to 50,000 blocks of up to 4 MB each, for a total size of slightly more than 195 GB (4 MB X 50,000).Page blobs can be up to 1 TB in size, and are more efficient for frequent read/write operations. Azure Virtual Machines use page blobs as OS and data disks.

Prerequisites:

• Microsoft Visual Studio
• Azure Storage Client Library for .NET
• Azure Configuration Manager for .NET
• An Azure storage account

Create a Windows console application project

In Visual Studio, create a new Windows console application, as shown:

All of the code examples in this tutorial can be added to the Main() method in program.cs in your console application.

Note that you can use the Azure Storage Client Library from any type of .NET application, including an Azure cloud service, an Azure web app, a desktop application, or a mobile application. In this guide, we use a console application for simplicity.

Use NuGet to install the required packages

There are two packages that you’ll need to install to your project to complete this tutorial:

• Microsoft Azure Storage Client Library for .NET: This package provides programmatic access to data resources in your storage account.
• Microsoft Azure Configuration Manager library for .NET: This package provides a class for parsing a connection string from a configuration file, regardless of where your application is running.

You can use NuGet to obtain both packages. Follow these steps:

1. Right-click your project in Solution Explorer and choose Manage NuGet Packages.
2. Search online for “WindowsAzure.Storage” and click Install to install the Storage Client Library and its dependencies.
3. Search online for “ConfigurationManager” and click Install to install the Azure Configuration Manager.

Determine your target environment

You have two environment options for running the examples in this guide:

• You can run your code against an Azure Storage account in the cloud.
• You can run your code against the Azure storage emulator. The storage emulator is a local environment that emulates an Azure Storage account in the cloud. The emulator is a free option for testing and debugging your code while your application is under development. The emulator uses a well-known account and key.

If you are targeting a storage account in the cloud, copy the primary access key for your storage account from the Azure Portal.

Configure your storage connection string

The Azure Storage Client Library for .NET supports using a storage connection string to configure endpoints and credentials for accessing storage services. The best way to maintain your storage connection string is in a configuration file.

To configure your connection string, open the app.config file from Solution Explorer in Visual Studio. Add the contents of the <appSettings> element shown below. Replace account-namewith the name of your storage account, and account-key with your account access key:

<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
    </startup>
    <appSettings>
        <add key="StorageConnectionString" 
         value="DefaultEndpointsProtocol=https;AccountName=account-name;
         AccountKey=account-key" />
    </appSettings>
</configuration>

For example, your configuration setting will be similar to:

<add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;
AccountName=storagesample;
AccountKey=nYV0gln6fT7mvY+rxu2iWAEyzPKITGkhM88J8HUoyofvK7C6fHcZc2kRZp6cKgYRUM74lHI84L50Iau1+9hPjB==" />

To target the storage emulator, you can use a shortcut that maps to the well-known account name and key. In that case, your connection string setting will be:

<add key="StorageConnectionString" value="UseDevelopmentStorage=true;" />

Add namespace declarations

Add the following using statements to the top of the program.cs file:

using Microsoft.Azure; // Namespace for CloudConfigurationManager
using Microsoft.WindowsAzure.Storage; // Namespace for CloudStorageAccount
using Microsoft.WindowsAzure.Storage.Blob; // Namespace for Blob storage types

Parse the connection string

The Microsoft Azure Configuration Manager Library for .NET provides a class for parsing a connection string from a configuration file. The CloudConfigurationManager class parses configuration settings regardless of whether the client application is running on the desktop, on a mobile device, in an Azure virtual machine, or in an Azure cloud service.

Here’s an example that shows how to use retrieve a connection string from a configuration file:

// Parse the connection string and return a reference to the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
    CloudConfigurationManager.GetSetting("StorageConnectionString"));

Using Azure Configuration Manager is optional.

Create the Blob service client

The CloudBlobClient class enables you to retrieve containers and blobs stored in Blob storage. Here’s one way to create the service client:

CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

Now you are ready to write code that reads data from and writes data to Blob storage.

Create a container

Every blob in Azure storage must reside in a container. The container forms part of the blob name. For example, mycontainer is the name of the container in these sample blob URIs:

https://storagesample.blob.core.windows.net/mycontainer/blob1.txt
https://storagesample.blob.core.windows.net/mycontainer/photos/myphoto.jpg

A container name must be a valid DNS name, conforming to the following naming rules:

1. Container names must start with a letter or number, and can contain only letters, numbers, and the dash (-) character.
2. Every dash (-) character must be immediately preceded and followed by a letter or number; consecutive dashes are not permitted in container names.
3. All letters in a container name must be lowercase.
4. Container names must be from 3 through 63 characters long.

This example shows how to create a container if it does not already exist:

// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
    CloudConfigurationManager.GetSetting("StorageConnectionString"));

// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");

// Create the container if it doesn't already exist.
container.CreateIfNotExists();

By default, the new container is private, meaning that you must specify your storage access key to download blobs from this container. If you want to make the files within the container available to everyone, you can set the container to be public using the following code:

container.SetPermissions(
    new BlobContainerPermissions { PublicAccess = 
        BlobContainerPublicAccessType.Blob });

Anyone on the Internet can see blobs in a public container, but you can modify or delete them only if you have the appropriate account access key or a shared access signature.

Upload a blob into a container

Azure Blob Storage supports block blobs and page blobs. In the majority of cases, block blob is the recommended type to use.

To upload a file to a block blob, get a container reference and use it to get a block blob reference. Once you have a blob reference, you can upload any stream of data to it by calling the UploadFromStream method. This operation will create the blob if it didn’t previously exist, or overwrite it if it does exist.

The following example shows how to upload a blob into a container and assumes that the container was already created.

// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
    CloudConfigurationManager.GetSetting("StorageConnectionString"));

// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");

// Retrieve reference to a blob named "myblob".
CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob");

// Create or overwrite the "myblob" blob with contents from a local file.
using (var fileStream = System.IO.File.OpenRead(@"path\myfile"))
{
    blockBlob.UploadFromStream(fileStream);
}

List the blobs in a container

To list the blobs in a container, first get a container reference. You can then use the container’sListBlobs method to retrieve the blobs and/or directories within it. To access the rich set of properties and methods for a returned IListBlobItem, you must cast it to a CloudBlockBlob,CloudPageBlob, or CloudBlobDirectory object. If the type is unknown, you can use a type check to determine which to cast it to. The following code demonstrates how to retrieve and output the URI of each item in the photos container:

// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
    CloudConfigurationManager.GetSetting("StorageConnectionString"));

// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("photos");

// Loop over items within the container and output the length and URI.
foreach (IListBlobItem item in container.ListBlobs(null, false))
{
    if (item.GetType() == typeof(CloudBlockBlob))
    {
        CloudBlockBlob blob = (CloudBlockBlob)item;

        Console.WriteLine("Block blob of length {0}: {1}", blob.Properties.Length,
                               blob.Uri);

    }
    else if (item.GetType() == typeof(CloudPageBlob))
    {
        CloudPageBlob pageBlob = (CloudPageBlob)item;

        Console.WriteLine("Page blob of length {0}: {1}", pageBlob.Properties.
                  Length, pageBlob.Uri);

    }
    else if (item.GetType() == typeof(CloudBlobDirectory))
    {
        CloudBlobDirectory directory = (CloudBlobDirectory)item;

        Console.WriteLine("Directory: {0}", directory.Uri);
    }
}

As shown above, you can name blobs with path information in their names. This creates a virtual directory structure that you can organize and traverse as you would a traditional file system. Note that the directory structure is virtual only – the only resources available in Blob storage are containers and blobs. However, the storage client library offers a CloudBlobDirectory object to refer to a virtual directory and simplify the process of working with blobs that are organized in this way.

For example, consider the following set of block blobs in a container named photos:

photo1.jpg
2010/architecture/description.txt
2010/architecture/photo3.jpg
2010/architecture/photo4.jpg
2011/architecture/photo5.jpg
2011/architecture/photo6.jpg
2011/architecture/description.txt
2011/photo7.jpg

When you call ListBlobs on the ‘photos’ container (as in the above sample), a hierarchical listing is returned. It contains both CloudBlobDirectory and CloudBlockBlob objects, representing the directories and blobs in the container, respectively. The resulting output looks like:

Directory: https://<accountname>.blob.core.windows.net/photos/2010/
Directory: https://<accountname>.blob.core.windows.net/photos/2011/
Block blob of length 505623: https://<accountname>.blob.core.windows.net/photos/photo1.jpg

Optionally, you can set the UseFlatBlobListing parameter of of the ListBlobs method to true. In this case, every blob in the container is returned as a CloudBlockBlob object. The call toListBlobs to return a flat listing looks like this:

// Loop over items within the container and output the length and URI.
foreach (IListBlobItem item in container.ListBlobs(null, true))
{
   ...
}

Block blob of length 4: https://<accountname>.blob.core.windows.net/photos/2010/architecture/description.txt
Block blob of length 314618: https://<accountname>.blob.core.windows.net/photos/2010/architecture/photo3.jpg
Block blob of length 522713: https://<accountname>.blob.core.windows.net/photos/2010/architecture/photo4.jpg
Block blob of length 4: https://<accountname>.blob.core.windows.net/photos/2011/architecture/description.txt
Block blob of length 419048: https://<accountname>.blob.core.windows.net/photos/2011/architecture/photo5.jpg
Block blob of length 506388: https://<accountname>.blob.core.windows.net/photos/2011/architecture/photo6.jpg
Block blob of length 399751: https://<accountname>.blob.core.windows.net/photos/2011/photo7.jpg
Block blob of length 505623: https://<accountname>.blob.core.windows.net/photos/photo1.jpg

In this article, let’s have look at some basic concepts about Azure blobs. We will also see how to use .NET SDK with Blob storage.

What is Blob ?

Blob stands for Binary Large Object. Blob storage can be used for storing the unstructured data.

The unstructured data may include images, video files, audio files, VHDs, or even regular text / Word/ Excel/ PowerPoint files or even PDF files.

But you may have a question now.

When To Prefer Blob Storage over Azure Files ?

Azure Files is a storage service which also allows you to store the files. The file share can be accessed using HTTP / HTTPS REST APIs from anywhere in the world.

You can also store any type of files in Azure blob storage. That also can be accessed over HTTP / HTTPs from anywhere in the world.

So, the obvious query is when to use Azure Blob over Azure Files ?

Azure files also supports SMB protocol and you can mount the File Shre as a drive on your virtual machine or on premises machines.

If your application DOES NEED to access files/fas a network drive, then I guess it would make sense to use Azure Files than blob. But if it is ok to always access files on HTTP / HTTPs then you may want to choose blob storage.

Blob Storage Resources

There are three important terms here: storage account, blob containers and blobs.

Storage accounts provide unique namespace and domain name for your blob.

Inside a storage account, you can create blob containers to hold the blob data. A container organizes a set of blobs, similar to a directory in a file system. It is the top level directory for all its contents.

A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs.

Types of blobs

There are three types of blobs”

• Block Blob: the blob type which can hold either text or binary data. Each blob is like a block and it can be managed independently.
• Append Blob: This is similar to block blob, except it is further optimized for appending scenarios. Every blob consists of multiple blocks and append operation can append further blocks. This type is ideal for logging scenarios.
• Page Blob: This is to store random access files. It is mainly used for storing VHD files of virtual machines.

Create Blob Storage

You need to create a storage account before creating a blob storage instance. We already have seen how to create the storage account instance in one of my previous blogs.

Creating blob storage is as easy as creating the folder.

If you are logged in to Azure Portal, you can navigate to the Azure Storage account and then select blob service. The right side panel will show a button to add containers.

When you click on + Container button, a popup will be shown for adding new container. In that popup, you can provide the blob container name and the access level.

Access level can be

• Private – no anonymous access meaning you cannot access blob and cannot list container items anonymously. You must be a valid user to perform these operations. This is the default access level.
• Blob – anonymous read access for blobs only meaning you can read the blob anonymously, but you cannot list the contents of blob container anonymously.
• Container – anonymous read access for containers and blobs meaning you can list the contents of container or read the blob anonymously

Let’s keep the default access level for this demo. Provide “firstcontainer” as the name of container and click on OK. The container would be created instantly.

Картинка с сайта: thecodeblogger.com

Blob Operations using Azure CLI

Below code shows a sample script to create the container using Azure CLI and some basic operations on blob storage.

# TO create storage account
az storage account create      \
    --name storageaccountname  \
    --resource-group SamplesRG \
    --location WestEurope      \
    --sku Standard_ZRS         \
    --encryption blob

# To export the environment variables
export AZURE_STORAGE_ACCOUNT="<account-name>"
export AZURE_STORAGE_KEY="<account-key>"

# To create the blob container
az storage container create --name sample-container

# To upload a file
az storage blob upload              \
    --container-name firstcontainer \
    --name firstfile                    \
    --file firstfile.txt

# To List contents of container
az storage blob list \
    --container-name firstcontainer \
    --output table

# To download the blob contents
az storage blob download \
    --container-name sample-container \
    --name firstfile\
    --file destination-path

# AzCopy utility to upload data to storage
azcopy --source /mnt/myfiles \
    --destination https://<account>.blob.core.windows.net/firstcontainer \
    --dest-key <account-key> \
    --include "firstfile.txt"

Blob Operations Using .NET SDK

.NET SDK is available in two versions as of now – v11 and v12. Below steps are useful if you want to use v12 SDK.

The steps for v11 are also almost same, the package name is different. It is Microsoft.Azure.Storage.Blob.

Below are the important classes which we are going to use in this section:

Use the following .NET classes to interact with these resources:

• BlobServiceClient: to manage Azure storage resources and containers.
• BlobContainerClient: to manipulate Azure Storage containers and their blobs.
• BlobClient: to manipulate Azure Storage blobs.
• BlobDownloadInfo: represents the properties and content returned from downloading a blob.

For using .NET SDK, below are the steps:

• Get connection string of Azure blob
• Create the .NET project / solution
• Add reference to NuGet package Azure.Storage.Blobs
• Perform operations like create container, upload/download blob, etc.

Get Connection String

Sign in to Azure Portal and navigate to storage account. In the Settings section of the storage account overview, select Access keys.

Here, you can view the account keys and the complete connection string for each key. Find the connection string under any key (let’s say Key 1).

Add NuGet Package

For installing NuGet package, you can add reference to the NuGet package using Add Reference.

Alternatively, you can execute below command on NuGet Package Manager console.

Install-Package Azure.Storage.Blobs

C# Code

The BlobServiceClient object can be used to connect with Blob service. Below code calls CreateBlobContainerAsync method to create a new container with specified name.

var blobServiceClient = new BlobServiceClient(connectionString);
var containerName = "<some-unique-name>";
var containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName);

You can get reference to new container by using GetBlobClient method of BlobContainerClient. Then you can use stream to upload the file to the blob container as shown in below code.

// Get a reference to a blob
BlobClient blobClient = containerClient.GetBlobClient(fileName);

// Open the file and upload its data
using FileStream uploadFileStream = File.OpenRead(fileName);

// Upload the file
await blobClient.UploadAsync(uploadFileStream);

// Close the stream
uploadFileStream.Close();

Below code is to list all blobs from the container.

// List all blobs in container
await foreach (BlobItem blobItem in containerClient.GetBlobsAsync())
{
    Console.WriteLine("\t" + blobItem.Name);
}

The BlobDownloadInfo can be used to download the file as shown in below code.

// Download the blob's contents and save it to a file
var download = await blobClient.DownloadAsync();

using FileStream downloadFileStream = File.OpenWrite(downloadFilePath);
await download.Content.CopyToAsync(downloadFileStream);
downloadFileStream.Close();

To delete the container, you can just call delete method on the blob container.

await containerClient.DeleteAsync();

I hope this helps to get started on Azure Blob storage. Let me know your thoughts.