Instrukcja do tego ćwiczenia Pobierz notatnik (.ipynb)

Working with Datastores¶

In previous labs, you connected to an Azure ML workspace using the SDK v2 and ran a simple job based on data in a local CSV file. Now it's time to work with data stored in the cloud.

Important: The code in this notebook assumes that you have completed the first two tasks in Lab 4A. If you have not done so, go and do it now!

Connect to Your Workspace¶

The first thing you need to do is to connect to your workspace using the Azure ML SDK v2.

Note: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate.

In [ ]:

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
ml_client = MLClient.from_config(credential=credential)
print(f"Ready to use Azure ML to work with {ml_client.workspace_name}")

View Datastores in the Workspace¶

The workspace contains several datastores, including the aml_data datastore you created in the previous task.

Run the following code to retrieve the default datastore, and then list all of the datastores indicating which is the default.

In [ ]:

# Get the default datastore
default_ds = ml_client.datastores.get_default()

# List all datastores, indicating which is the default
for ds in ml_client.datastores.list():
    print(ds.name, "- Default =", ds.name == default_ds.name)

Get a Datastore to Work With¶

You want to work with the aml_data datastore, so you need to get it by name:

In [ ]:

aml_datastore = ml_client.datastores.get(name="aml_data")
print(f"{aml_datastore.name}: {aml_datastore.type} ({aml_datastore.account_name})")

Where the uploaded files actually land¶

You have picked a datastore, so it is time to put data in it. Here comes the first surprise.

In SDK v2, uploading files from disk happens as a side effect of registering a data asset. When a Data object is given a local path, the SDK uploads the files for you - but always to the default datastore. The Data class has no parameter for choosing a different one. This is not an oversight: SDK v1 had Datastore.upload_files(), and v2 has no equivalent.

What this means in practice: to place files from disk into a specific datastore, you use azcopy or the Data → Datastores → Browse page in Azure Machine Learning studio. From the SDK you can, however, write a job output to any datastore you like - and that is exactly what we will do with aml_data later in this lab.

In [ ]:

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

diabetes_data_folder = Data(
    path="./data",
    type=AssetTypes.URI_FOLDER,
    description="Diabetes data files (folder)",
    name="diabetes_data_folder",
)

diabetes_data_folder = ml_client.data.create_or_update(diabetes_data_folder)
print(f"Data asset registered: {diabetes_data_folder.name} (version {diabetes_data_folder.version})")

Train a Model Using Data From a Datastore¶

You pass a registered data asset to a job as an input: an Input object that points to the data asset by name and version. Azure ML then mounts or downloads the data to the compute target so the script can read it, regardless of where the job actually runs.

More Information: For more details about working with data, see the Azure ML documentation.

In [ ]:

from azure.ai.ml import Input
from azure.ai.ml.constants import AssetTypes

data_input = Input(type=AssetTypes.URI_FOLDER, path=f"azureml:{diabetes_data_folder.name}:{diabetes_data_folder.version}")
print(data_input)

To use the data input in a training script, you must define a parameter for it. Run the following two code cells to create:

A folder named diabetes_training_from_datastore
A script that trains a classification model by using the training data in all of the CSV files in the folder referenced by the data input passed to it.

In [ ]:

import os

# Create a folder for the job files
experiment_folder = 'diabetes_training_from_datastore'
os.makedirs(experiment_folder, exist_ok=True)
print(experiment_folder, 'folder created.')

In [ ]:

%%writefile $experiment_folder/diabetes_training.py
# Import libraries
import os
import argparse
import mlflow
import pandas as pd
import numpy as np
import joblib
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve

# Get parameters
parser = argparse.ArgumentParser()
parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')
parser.add_argument('--data-folder', type=str, dest='data_folder', help='data folder input')
parser.add_argument('--output-folder', type=str, dest='output_folder',
                    required=True, help='folder for the combined data')
args = parser.parse_args()
reg = args.reg_rate

# Start an MLflow run to log metrics (MLflow tracking is built into Azure ML v2 jobs)
mlflow.start_run()

# load the diabetes data from the data folder
data_folder = args.data_folder
print("Loading data from", data_folder)
# Load all files and concatenate their contents as a single dataframe
all_files = os.listdir(data_folder)
diabetes = pd.concat((pd.read_csv(os.path.join(data_folder, csv_file)) for csv_file in all_files))

# Separate features and labels
X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values

# Split data into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)

# Train a logistic regression model
print('Training a logistic regression model with regularization rate of', reg)
mlflow.log_metric('Regularization Rate', float(reg))
model = LogisticRegression(C=1/reg, solver="liblinear").fit(X_train, y_train)

# calculate accuracy
y_hat = model.predict(X_test)
acc = np.average(y_hat == y_test)
print('Accuracy:', acc)
mlflow.log_metric('Accuracy', float(acc))

# calculate AUC
y_scores = model.predict_proba(X_test)
auc = roc_auc_score(y_test, y_scores[:,1])
print('AUC: ' + str(auc))
mlflow.log_metric('AUC', float(auc))

os.makedirs('outputs', exist_ok=True)
# files saved in the outputs folder are automatically captured as job outputs
joblib.dump(value=model, filename='outputs/diabetes_model.pkl')

# Write the combined data to the folder pointed at by the job output.
# That output points at the aml_data datastore, so this is where the file lands.
os.makedirs(args.output_folder, exist_ok=True)
combined_path = os.path.join(args.output_folder, 'diabetes_combined.csv')
diabetes.to_csv(combined_path, index=False)
print('Combined data written to:', combined_path)

mlflow.end_run()

The script loads the training data from the data input passed to it as a parameter, so now you just need to set up the job to pass the data input when you submit it.

In [ ]:

from azure.ai.ml import command, Input, Output
from azure.ai.ml.constants import AssetTypes

job = command(
    code=experiment_folder,
    command="python diabetes_training.py --regularization 0.1 --data-folder ${{inputs.data_folder}} --output-folder ${{outputs.combined}}",
    inputs={
        "data_folder": Input(type=AssetTypes.URI_FOLDER, path=f"azureml:{diabetes_data_folder.name}:{diabetes_data_folder.version}")
    },
    outputs={
        # The output points at a specific datastore - this is how SDK v2
        # writes data where you want it, not where the default happens to be.
        "combined": Output(
            type=AssetTypes.URI_FOLDER,
            path="azureml://datastores/aml_data/paths/diabetes-combined/",
        ),
    },
    environment="azureml://registries/azureml/environments/sklearn-1.5/labels/latest",
    compute="aml-cluster",
    display_name="diabetes-training-datastore",
    experiment_name="diabetes-training",
)

# submit the job
returned_job = ml_client.jobs.create_or_update(job)
ml_client.jobs.stream(returned_job.name)

The first time the job runs, it may take some time to build the environment - subsequent runs will be quicker.

While (or after) the job runs, you can view its details in the Azure ML Studio web interface, including the Outputs + logs tab (look for user_logs/std_log.txt to verify that the data files were loaded), and you can write code to retrieve the metrics that were logged with MLflow:

Where to find the result: when the job finishes, open Data → Datastores → aml_data → Browse in Azure Machine Learning studio. The diabetes-combined folder holds diabetes_combined.csv - proof that the job wrote to the datastore you chose, not to the default one.

In [ ]:

import mlflow

mlflow.set_tracking_uri(ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri)
mlflow_run = mlflow.get_run(returned_job.name)

print("Metrics:")
for key, value in mlflow_run.data.metrics.items():
    print(key, value)

print(f"\nView run details in Studio: {returned_job.studio_url}")

Once again, you can register the model that was trained by the job.

In [ ]:

from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes

model = Model(
    path=f"azureml://jobs/{returned_job.name}/outputs/artifacts/paths/outputs/diabetes_model.pkl",
    name="diabetes_model",
    type=AssetTypes.CUSTOM_MODEL,
    description="Diabetes classification model",
    tags={"Training context": "Command job (using Datastore)"},
)
registered_model = ml_client.models.create_or_update(model)

# List the registered models
print("Registered Models:")
for m in ml_client.models.list(name="diabetes_model"):
    print(m.name, 'version:', m.version)
    for tag_name in m.tags:
        print('\t', tag_name, ':', m.tags[tag_name])

In this exercise, you've explored some options for working with data in the form of datastores.

Azure Machine Learning offers a further level of abstraction for data in the form of data assets, which you'll explore next.