Instrukcja do tego ćwiczenia Pobierz notatnik (.ipynb)

Working with Datasets¶

In the previous labs, you used a datastore to provide centralized, cloud-based data access. In this lab, you'll explore data assets (the SDK v2 successor to v1 datasets), a further abstraction that makes it easier to work with specific data for jobs and training.

Connect to Your Workspace¶

The first thing you need to do is to connect to your workspace using the Azure ML SDK v2.

Note: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate.

In [ ]:

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
ml_client = MLClient.from_config(credential=credential)
print(f"Ready to use Azure ML to work with {ml_client.workspace_name}")

Prepare Data¶

In the previous lab, you registered a uri_folder data asset. Data assets are usually (though not always) based on data uploaded to a datastore.

If you did not complete the previous lab, run the following code to register the two local CSV files as a data asset (if you did complete the previous lab, this just re-registers the same data, resulting in a new version of the same asset).

In [ ]:

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

diabetes_data_folder = Data(
    path="./data",
    type=AssetTypes.URI_FOLDER,
    description="Diabetes data files (folder)",
    name="diabetes_data_folder",
)
diabetes_data_folder = ml_client.data.create_or_update(diabetes_data_folder)
print(f"Data asset ready: {diabetes_data_folder.name} (version {diabetes_data_folder.version})")

Create a Tabular Data Asset¶

To work with tabular data in Azure ML you use an mltable: a table definition that describes how to read one or more delimited files as a single tabular dataset. Let's build an mltable from the diabetes data you uploaded, and view the first 20 records. In this case, the data is in structured CSV files, so we'll use mltable.from_delimited_files().

In [ ]:

# The mltable package together with its data-reading engine. Needed once per compute instance.
%pip install -q -U mltable "azureml-dataprep[pandas]"

In [ ]:

import mltable

# Create a table definition from the diabetes CSV files uploaded to the datastore (this may take a short while)
path = {"pattern": f"{diabetes_data_folder.path.rstrip('/')}/*.csv"}
tab_data_set = mltable.from_delimited_files(paths=[path])

# Display the first 20 rows as a Pandas dataframe
tab_data_set.to_pandas_dataframe().head(20)

As you can see in the code above, it's easy to materialize an mltable as a Pandas dataframe, enabling you to work with the data using common Python techniques.

Create a File Data Asset¶

The mltable you created lets you read all of the structured files it references as a single dataframe. This works well for tabular data, but in some machine learning scenarios you might need to work with unstructured data, or you may simply want to handle reading the data from files in your own code. To accomplish this, you can work with the same data as a uri_folder: a reference to a folder in a datastore, which you can browse and read using a filesystem-like API - the azureml-fsspec package.

In [ ]:

from azureml.fsspec import AzureMachineLearningFileSystem

# Create a filesystem view over the same folder (this may take a short while)
fs = AzureMachineLearningFileSystem(diabetes_data_folder.path)
file_data_set = fs.glob(f"{diabetes_data_folder.path.rstrip('/')}/*.csv")

# Print the files in the dataset
for file_path in file_data_set:
    print(file_path)

Register Data Assets¶

Now that you have created an mltable and a file-based view over the diabetes data, you can register them to make them easily accessible to any job run in the workspace.

You'll register the table as diabetes_mltable, and the folder as diabetes_file_dataset.

Why not diabetes_dataset: an asset of that name was created in Lab 1A through the studio wizard, which takes the older (v1) API path. SDK v2 can read such an asset, but it cannot add a new version to it - the attempt fails with Cannot create V2 Data Version in V1 Data Container. Data assets in Azure ML cannot be deleted (this is what guarantees experiment reproducibility), so that name stays claimed by v1 permanently. Assets registered from code therefore get a name of their own.

In [ ]:

# Save the table definition locally, then register it as a new version of diabetes_mltable
tab_data_set.save("./diabetes-mltable", colocated=True, overwrite=True)

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

tabular_data_asset = Data(
    path="./diabetes-mltable",
    type=AssetTypes.MLTABLE,
    description="diabetes data (tabular, both CSV files)",
    name="diabetes_mltable",
    tags={"format": "CSV"},
)
tabular_data_asset = ml_client.data.create_or_update(tabular_data_asset)

# Register the file data asset
file_data_asset = Data(
    path="./data",
    type=AssetTypes.URI_FOLDER,
    description="diabetes files",
    name="diabetes_file_dataset",
    tags={"format": "CSV"},
)
file_data_asset = ml_client.data.create_or_update(file_data_asset)

print(f"Registered {tabular_data_asset.name} (version {tabular_data_asset.version}) and {file_data_asset.name} (version {file_data_asset.version})")

You can view and manage data assets on the Data assets page for your workspace in the Azure ML Studio web interface. You can also get a list of data assets from the SDK:

In [ ]:

print("Data assets in the workspace:")
for data_asset in ml_client.data.list():
    print(f"\t{data_asset.name}")

# The version and type belong to a specific version of an asset, not to its name
for name in ["diabetes_mltable", "diabetes_file_dataset"]:
    latest = ml_client.data.get(name=name, label="latest")
    print(f"\n{latest.name}: latest version {latest.version}, type {latest.type}")

Registering the asset above created diabetes_mltable as version 1. Run that cell again and you get version 2, while version 1 remains available unchanged. The ability to version data assets enables you to redefine them without breaking existing jobs or pipelines that rely on previous definitions. By default, ml_client.data.get() without a version returns the latest version, but you can retrieve a specific version of a data asset by specifying the version number, like this:

dataset_v1 = ml_client.data.get(name="diabetes_mltable", version="1")

Train a Model from a Tabular Data Asset¶

Now that you have data assets, you're ready to start training models from them. You pass a data asset to a job as an input, using the Input class.

Run the following two code cells to create:

A folder named diabetes_training_from_tab_dataset
A script that trains a classification model by using a tabular (mltable) data asset that is passed to it as an input.

In [ ]:

import os

# Create a folder for the job files
experiment_folder = 'diabetes_training_from_tab_dataset'
os.makedirs(experiment_folder, exist_ok=True)
print(experiment_folder, 'folder created')

In [ ]:

%%writefile $experiment_folder/diabetes_training.py
# Import libraries
import argparse
import os
import mlflow
import mltable
import pandas as pd
import numpy as np
import joblib
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve

# Set regularization hyperparameter (passed as an argument to the script)
parser = argparse.ArgumentParser()
parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')
parser.add_argument('--training-data', type=str, dest='training_data', help='path to the training data mltable')
args = parser.parse_args()
reg = args.reg_rate

# Start an MLflow run to log metrics (MLflow tracking is built into Azure ML v2 jobs)
mlflow.start_run()

# load the diabetes data (passed as an mltable input)
print("Loading Data...")
tbl = mltable.load(args.training_data)
diabetes = tbl.to_pandas_dataframe()

# Separate features and labels
X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values

# Split data into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)

# Train a logistic regression model
print('Training a logistic regression model with regularization rate of', reg)
mlflow.log_metric('Regularization Rate', float(reg))
model = LogisticRegression(C=1/reg, solver="liblinear").fit(X_train, y_train)

# calculate accuracy
y_hat = model.predict(X_test)
acc = np.average(y_hat == y_test)
print('Accuracy:', acc)
mlflow.log_metric('Accuracy', float(acc))

# calculate AUC
y_scores = model.predict_proba(X_test)
auc = roc_auc_score(y_test, y_scores[:,1])
print('AUC: ' + str(auc))
mlflow.log_metric('AUC', float(auc))

os.makedirs('outputs', exist_ok=True)
# files saved in the outputs folder are automatically captured as job outputs
joblib.dump(value=model, filename='outputs/diabetes_model.pkl')

mlflow.end_run()

Now you can create a job to run the script, and define a named input for the training data asset, which is read by the script.

Note: The mltable package is used to load the data asset in the script, so it must be available in the job's execution environment. Curated environments - including the sklearn environment used in earlier labs - do not include it. That's why the cell below defines a custom environment containing it. The first run builds a container image from that definition, which takes a few minutes; later jobs reuse the image.

In [ ]:

from azure.ai.ml.entities import Environment

# The script reads its input with the mltable package, so that package must be
# present in the job's environment. The curated sklearn environment does not
# include it, so we define our own.
conda_spec = {
    "name": "diabetes-mltable-env",
    "channels": ["conda-forge"],
    "dependencies": [
        "python=3.10",
        "scikit-learn",
        "pandas",
        "numpy",
        "pip",
        {
            "pip": [
                "mltable",
                "azureml-dataprep[pandas]",
                # The script saves the model with mlflow.sklearn, so the full mlflow
                # package is needed. Pinned to the upper bound that azureml-mlflow
                # supports - a newer one breaks artifact logging.
                "mlflow<=3.15.0",
                "azureml-mlflow",
            ]
        },
    ],
}

mltable_env = Environment(
    name="diabetes-mltable-env",
    description="Environment with the mltable package for reading tabular data assets",
    conda_file=conda_spec,
    image="mcr.microsoft.com/azureml/openmpi5.0-ubuntu24.04",
)

# Register in the workspace. Passing the object straight to a job would give an
# anonymous environment - nameless, and separate for every job.
mltable_env = ml_client.environments.create_or_update(mltable_env)

print(f"{mltable_env.name}:{mltable_env.version} - environment registered.")

In [ ]:

from azure.ai.ml import command, Input
from azure.ai.ml.constants import AssetTypes

job = command(
    code=experiment_folder,
    command="python diabetes_training.py --regularization 0.1 --training-data ${{inputs.training_data}}",
    inputs={
        "training_data": Input(type=AssetTypes.MLTABLE, path=f"azureml:{tabular_data_asset.name}:{tabular_data_asset.version}", mode="ro_mount")
    },
    environment=f"{mltable_env.name}:{mltable_env.version}",
    compute="aml-cluster",
    display_name="diabetes-training-tabular",
    experiment_name="diabetes-training",
)

# submit the job
returned_job = ml_client.jobs.create_or_update(job)
ml_client.jobs.stream(returned_job.name)

The first time the job runs, it may take some time to build the environment - subsequent runs will be quicker.

When the job has completed, you can view its details in the Azure ML Studio web interface, including the Outputs + logs tab and the metrics logged by the run, and you can write code to retrieve them:

In [ ]:

import mlflow

mlflow.set_tracking_uri(ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri)
mlflow_run = mlflow.get_run(returned_job.name)

print("Metrics:")
for key, value in mlflow_run.data.metrics.items():
    print(key, value)

print(f"\nView run details in Studio: {returned_job.studio_url}")

The model we trained is saved as the diabetes_model.pkl file in the outputs folder, so you can register it.

In [ ]:

from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes

model = Model(
    path=f"azureml://jobs/{returned_job.name}/outputs/artifacts/paths/outputs/diabetes_model.pkl",
    name="diabetes_model",
    type=AssetTypes.CUSTOM_MODEL,
    tags={"Training context": "Command job (tabular data asset)"},
)
ml_client.models.create_or_update(model)

for m in ml_client.models.list(name="diabetes_model"):
    print(m.name, 'version:', m.version)
    for tag_name in m.tags:
        print('\t', tag_name, ':', m.tags[tag_name])

Train a Model from a File Data Asset¶

You've seen how to train a model using a tabular (mltable) data asset; but what about a uri_folder file data asset?

When you use a uri_folder input, the value passed to the script is a mount point (or downloaded folder) containing file paths. How you read the data from these files depends on the kind of data in them and what you want to do with it. In the case of the diabetes CSV files, you can use the Python glob module to create a list of the files in the folder, and read them all into Pandas dataframes that are concatenated into a single dataframe.

Run the following two code cells to create:

A folder named diabetes_training_from_file_dataset
A script that trains a classification model by using a uri_folder data asset that is passed to it as an input.

In [ ]:

import os

# Create a folder for the job files
experiment_folder = 'diabetes_training_from_file_dataset'
os.makedirs(experiment_folder, exist_ok=True)
print(experiment_folder, 'folder created')

In [ ]:

%%writefile $experiment_folder/diabetes_training.py
# Import libraries
import argparse
import os
import glob
import mlflow
import pandas as pd
import numpy as np
import joblib
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve

# Set regularization hyperparameter (passed as an argument to the script)
parser = argparse.ArgumentParser()
parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')
parser.add_argument('--training-data', type=str, dest='training_data', help='path to the folder of training data files')
args = parser.parse_args()
reg = args.reg_rate

# Start an MLflow run to log metrics (MLflow tracking is built into Azure ML v2 jobs)
mlflow.start_run()

# load the diabetes dataset
print("Loading Data...")
data_path = args.training_data
all_files = glob.glob(data_path + "/*.csv")
diabetes = pd.concat((pd.read_csv(f) for f in all_files))

# Separate features and labels
X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values

# Split data into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)

# Train a logistic regression model
print('Training a logistic regression model with regularization rate of', reg)
mlflow.log_metric('Regularization Rate', float(reg))
model = LogisticRegression(C=1/reg, solver="liblinear").fit(X_train, y_train)

# calculate accuracy
y_hat = model.predict(X_test)
acc = np.average(y_hat == y_test)
print('Accuracy:', acc)
mlflow.log_metric('Accuracy', float(acc))

# calculate AUC
y_scores = model.predict_proba(X_test)
auc = roc_auc_score(y_test, y_scores[:,1])
print('AUC: ' + str(auc))
mlflow.log_metric('AUC', float(auc))

os.makedirs('outputs', exist_ok=True)
# files saved in the outputs folder are automatically captured as job outputs
joblib.dump(value=model, filename='outputs/diabetes_model.pkl')

mlflow.end_run()

Next, you need to choose how the input is delivered to the compute target. For large volumes of data, you'd generally use mode="ro_mount" to stream the files directly from storage; but for a small dataset like this one, mode="download" (the SDK v2 equivalent of the v1 as_download option) works well too.

In [ ]:

from azure.ai.ml import command, Input
from azure.ai.ml.constants import AssetTypes

job = command(
    code=experiment_folder,
    command="python diabetes_training.py --regularization 0.1 --training-data ${{inputs.training_data}}",
    inputs={
        "training_data": Input(type=AssetTypes.URI_FOLDER, path=f"azureml:{file_data_asset.name}:{file_data_asset.version}", mode="download")
    },
    environment="azureml://registries/azureml/environments/sklearn-1.5/labels/latest",
    compute="aml-cluster",
    display_name="diabetes-training-file",
    experiment_name="diabetes-training",
)

# submit the job
returned_job = ml_client.jobs.create_or_update(job)
ml_client.jobs.stream(returned_job.name)

When the job has completed, you can view its details in the Azure ML Studio web interface, including the Outputs + logs tab, to verify that the file data asset was processed and the data files were read; and you can write code to retrieve the logged metrics:

In [ ]:

import mlflow

mlflow.set_tracking_uri(ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri)
mlflow_run = mlflow.get_run(returned_job.name)

print("Metrics:")
for key, value in mlflow_run.data.metrics.items():
    print(key, value)

print(f"\nView run details in Studio: {returned_job.studio_url}")

Once again, let's register the model that we trained.

In [ ]:

from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes

model = Model(
    path=f"azureml://jobs/{returned_job.name}/outputs/artifacts/paths/outputs/diabetes_model.pkl",
    name="diabetes_model",
    type=AssetTypes.CUSTOM_MODEL,
    tags={"Training context": "Command job (file data asset)"},
)
ml_client.models.create_or_update(model)

for m in ml_client.models.list(name="diabetes_model"):
    print(m.name, 'version:', m.version)
    for tag_name in m.tags:
        print('\t', tag_name, ':', m.tags[tag_name])

More Information: For more information about reading and writing data in jobs, see Read and write data in a job in the Azure ML documentation.