Set Up the GPU Droplet for BAGEL

apt install python3-pip python3.10-venv
pip install huggingface-hub wheel jupyter
git clone https://github.com/ByteDance-Seed/Bagel
cd Bagel/
vim requirements.txt

pip install -r requirements.txt
pip install git+https://github.com/Dao-AILab/flash-attention

huggingface-cli login

##After entering your access token

huggingface-cli download ByteDance-Seed/BAGEL-7B-MoT --local-folder ./BAGEL-7B-MoT

from huggingface_hub import snapshot_download

save_dir = "./BAGEL-7B-MoT"
repo_id = "ByteDance-Seed/BAGEL-7B-MoT"
cache_dir = save_dir + "/cache"

snapshot_download(cache_dir=cache_dir,
        local_dir=save_dir,
        repo_id=repo_id,
        local_dir_use_symlinks=False,
        resume_download=True,
        allow_patterns=["*.json", "*.safetensors", "*.bin", "*.py", "*.md", "*.txt"],
        )

jupyter lab --allow-root

import os
from copy import deepcopy
from typing import (
    Any,
    AsyncIterable,
    Callable,
    Dict,
    Generator,
    List,
    NamedTuple,
    Optional,
    Tuple,
    Union,
)
import requests
from io import BytesIO

from PIL import Image
import torch
from accelerate import infer_auto_device_map, load_checkpoint_and_dispatch, init_empty_weights

from data.transforms import ImageTransform
from data.data_utils import pil_img2rgb, add_special_tokens
from modeling.bagel import (
    BagelConfig, Bagel, Qwen2Config, Qwen2ForCausalLM, SiglipVisionConfig, SiglipVisionModel
)
from modeling.qwen2 import Qwen2Tokenizer
from modeling.bagel.qwen2_navit import NaiveCache
from modeling.autoencoder import load_ae
from safetensors.torch import load_file

model_path = "./BAGEL-7B-MoT"  
# LLM config preparing
llm_config = Qwen2Config.from_json_file(os.path.join(model_path, "llm_config.json"))
llm_config.qk_norm = True
llm_config.tie_word_embeddings = False
llm_config.layer_module = "Qwen2MoTDecoderLayer"

# ViT config preparing
vit_config = SiglipVisionConfig.from_json_file(os.path.join(model_path, "vit_config.json"))
vit_config.rope = False
vit_config.num_hidden_layers = vit_config.num_hidden_layers - 1

# VAE loading
vae_model, vae_config = load_ae(local_path=os.path.join(model_path, "ae.safetensors"))

# Bagel config preparing
config = BagelConfig(
    visual_gen=True,
    visual_und=True,
    llm_config=llm_config, 
    vit_config=vit_config,
    vae_config=vae_config,
    vit_max_num_patch_per_side=70,
    connector_act='gelu_pytorch_tanh',
    latent_patch_size=2,
    max_latent_size=64,
)

with init_empty_weights():
    language_model = Qwen2ForCausalLM(llm_config)
    vit_model = SiglipVisionModel(vit_config)
    model = Bagel(language_model, vit_model, config)
    model.vit_model.vision_model.embeddings.convert_conv2d_to_linear(vit_config, meta=True)

# Tokenizer Preparing
tokenizer = Qwen2Tokenizer.from_pretrained(model_path)
tokenizer, new_token_ids, _ = add_special_tokens(tokenizer)

# Image Transform Preparing
vae_transform = ImageTransform(1024, 512, 16)
vit_transform = ImageTransform(980, 224, 14)

max_mem_per_gpu = "80GiB"  # Modify it according to your GPU setting

device_map = infer_auto_device_map(
    model,
    max_memory={i: max_mem_per_gpu for i in range(torch.cuda.device_count())},
    no_split_module_classes=["Bagel", "Qwen2MoTDecoderLayer"],
)
print(device_map)

same_device_modules = [
    'language_model.model.embed_tokens',
    'time_embedder',
    'latent_pos_embed',
    'vae2llm',
    'llm2vae',
    'connector',
    'vit_pos_embed'
]

if torch.cuda.device_count() == 1:
    first_device = device_map.get(same_device_modules[0], "cuda:0")
    for k in same_device_modules:
        if k in device_map:
            device_map[k] = first_device
        else:
            device_map[k] = "cuda:0"
else:
    first_device = device_map.get(same_device_modules[0])
    for k in same_device_modules:
        if k in device_map:
            device_map[k] = first_device

model = load_checkpoint_and_dispatch(
    model,
    checkpoint=model_path+"/ema.safetensors"),
    device_map=device_map,
    offload_buffers=True,
    dtype=torch.bfloat16,
)

model = model.eval()
print('Model loaded')

from inferencer import InterleaveInferencer

inferencer = InterleaveInferencer(
        model=model,
        vae_model=vae_model,
        tokenizer=tokenizer,
        vae_transform=vae_transform,
        vit_transform=vit_transform,
        new_token_ids=new_token_ids
)

import random
import numpy as np

seed = 42
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
        torch.cuda.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Generate images with BAGEL

inference_hyper=dict(
        cfg_text_scale=4.0,
        cfg_img_scale=1.0,
        cfg_interval=[0.4, 1.0],
        timestep_shift=3.0,
        num_timesteps=50,
        cfg_renorm_min=1.0,
        cfg_renorm_type="global",
)

prompt = '''draw the famous actor Keanu Reeves with long hair and a beard eating Ramen noodles while sitting next to an anthromorphic bear wearing a bowtie, both laughing, drawing in anime style'''
print(prompt)
print('-' * 10)
output_dict = inferencer(text=prompt, **inference_hyper)
display(output_dict['image'])

from PIL import Image
import PIL

output_dict['image'].save('output_img_gen.png')

inference_hyper=dict(
        max_think_token_n=1000,
        do_sample=False,
        # text_temperature=0.3,
        cfg_text_scale=4.0,
        cfg_img_scale=1.0,
        cfg_interval=[0.4, 1.0],
        timestep_shift=3.0,
        num_timesteps=50,
        cfg_renorm_min=1.0,
        cfg_renorm_type="global",
)
prompt = '''draw the famous actor Keanu Reeves with long hair and a beard eating Ramen noodles while sitting next to an anthromorphic bear wearing a bowtie, Keanu is wearing a shirt that says "DigitalOean", both are laughing, drawing in anime style'''
print(prompt)
print('-' * 10)
output_dict = inferencer(text=prompt, think=True, **inference_hyper)
print(output_dict['text'])
display(output_dict['image'])

Image Editing

inference_hyper=dict(
        cfg_text_scale=4.0,
        cfg_img_scale=2.0,
        cfg_interval=[0.0, 1.0],
        timestep_shift=4.0,
        num_timesteps=50,
        cfg_renorm_min=1.0,
        cfg_renorm_type="text_channel",
)

image = Image.open('./output_img_gen_think.png')
prompt = 'make his shirt say "DIGITALOCEAN"'
display(image)
print(prompt)
print('-'*10)
output_dict = inferencer(image=image, text=prompt, **inference_hyper)
display(output_dict['image'])

Image Editing with Thinking

inference_hyper=dict(
        max_think_token_n=1000,
        do_sample=False,
        # text_temperature=0.3,
        cfg_text_scale=4.0,
        cfg_img_scale=2.0,
        cfg_interval=[0.4, 1.0],
        timestep_shift=3.0,
        num_timesteps=50,
        cfg_renorm_min=0.0,
        cfg_renorm_type="text_channel",
)

image = Image.open('./test_images/octupusy.jpg')
prompt = 'Could you display the sculpture that takes after this design?'

display(image)
print('-'*10)
output_dict = inferencer(image=image, text=prompt, think=True, **inference_hyper)
print(output_dict['text'])
display(output_dict['image'])

Image Understanding

inference_hyper=dict(
        max_think_token_n=1000,
        do_sample=False,
        # text_temperature=0.3,
)

image = Image.open('./test_images/meme.jpg')
prompt = "Can someone explain what’s funny about this meme??"

display(image)
print(prompt)
print('-'*10)
output_dict = inferencer(image=image, text=prompt, understanding_output=True, **inference_hyper)
print(output_dict['text'])

• MySQL installed and configured on your system.
• Basic knowledge of SQL syntax.
• Access to MySQL command line or MySQL Workbench.

CREATE TABLE table_name (
    column1_name data_type PRIMARY KEY,
    column2_name data_type,
    ...
);

CREATE TABLE users (
    id INT PRIMARY KEY,
    name VARCHAR(255),
    email VARCHAR(255)
);

MySQL Table Syntax without Primary Key

CREATE TABLE table_name (
    column1_name data_type,
    column2_name data_type,
    ...
);

CREATE DATABASE mydatabase;

USE mydatabase;

CREATE TABLE users (
    id INT PRIMARY KEY AUTO_INCREMENT,
    name VARCHAR(100),
    email VARCHAR(255) UNIQUE,
    registration_date DATE
);

• id: This column is defined as an integer (INT) and is set as the primary key of the table using PRIMARY KEY. The AUTO_INCREMENT attribute means that each time a new record is inserted into the table, the value of id will automatically increase by 1, starting from 1. This ensures that each record has a unique identifier.
  
• name and email: These columns are defined as variable-length strings using VARCHAR. The number in parentheses specifies the maximum length of the string that can be stored in each column. For name, the maximum length is 100 characters, and for email, it’s 255 characters. The UNIQUE attribute for email ensures that each email address in the table is unique and cannot be duplicated.
  
• registration_date: This column is defined as a DATE type, which is used to store dates. It will hold the date when each user registered.
  

• name: ‘John Doe’
• email: ‘john@example.com’
• registration_date: ‘2025-01-10’

INSERT INTO users (name, email, registration_date)
VALUES ('John Doe', 'john@example.com', '2025-01-10');

Inserting Multiple Rows

INSERT INTO users (name, email, registration_date)
VALUES
('Jane Smith', 'jane@example.com', '2025-01-11'),
('Emily Johnson', 'emily@example.com', '2025-01-12');

SELECT * FROM users;

Output
+----+------------+-------------------+----------------+
| id | name       | email             | registration_date |
+----+------------+-------------------+----------------+
|  1 | John Doe   | john@example.com | 2025-01-10      |
|  2 | Jane Smith | jane@example.com | 2025-01-11      |
|  3 | Emily Johnson | emily@example.com | 2025-01-12      |
+----+------------+-------------------+----------------+

UPDATE users SET email = 'john.doe@example.com' WHERE id = 1;

SELECT * FROM users;

Output
+----+------------+-------------------+----------------+
| id | name       | email             | registration_date |
+----+------------+-------------------+----------------+
|  1 | John Doe   | john.doe@example.com | 2025-01-10      |
|  2 | Jane Smith | jane@example.com | 2025-01-11      |
|  3 | Emily Johnson | emily@example.com | 2025-01-12      |
+----+------------+-------------------+----------------+

Inserting data for a blog, CRM, or e-commerce site

<?php
// Assuming $conn is a valid MySQL connection
if (isset($_POST['register'])) {
    $name = $_POST['name'];
    $email = $_POST['email'];
    $password = $_POST['password']; // Assuming password is hashed for security

    $query = "INSERT INTO users (name, email, password) VALUES (?, ?, ?)";
    $stmt = $conn->prepare($query);
    $stmt->bind_param("sss", $name, $email, $password);
    $stmt->execute();
    $stmt->close();
}
?>

Integration with backend workflows

const mysql = require('mysql');

// Assuming db is a valid MySQL connection
const insertCustomerInteraction = (customerID, interactionType, interactionDate) => {
    const query = "INSERT INTO customer_interactions (customer_id, interaction_type, interaction_date) VALUES (?, ?, ?)";
    db.query(query, [customerID, interactionType, interactionDate], (error, results, fields) => {
        if (error) throw error;
        console.log('Customer interaction inserted successfully');
    });
};

Table already exists

CREATE TABLE IF NOT EXISTS users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL
);

Incorrect data types

CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    age VARCHAR(3) NOT NULL // Incorrect data type for age, should be INT
);

INSERT INTO users (name, email, age) VALUES ('John Doe', 'john.doe@example.com', 'twenty-five'); // This will result in an error due to incorrect data type for age

CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    age INT NOT NULL // Correct data type for age
);

INSERT INTO users (name, email, age) VALUES ('John Doe', 'john.doe@example.com', 25); // Correct insertion with the right data type for age

Syntax errors

INSERT INTO users (name, email, age VALUES ('John Doe', 'john.doe@example.com', 25); // Missing closing parenthesis

INSERT INTO users (name, email, age) VALUES ('John Doe', 'john.doe@example.com', 25); // Correctly formatted SQL statement

INSERT

INSERT INTO users (name, email) VALUES ('John Doe', 'john.doe@example.com');

INSERT IGNORE

INSERT IGNORE INTO users (name, email) VALUES ('John Doe', 'john.doe@example.com');

REPLACE

REPLACE INTO users (name, email) VALUES ('John Doe', 'john.doe@example.com');

Comparison Table

• Use INSERT when you want to ensure that a row is inserted only if it doesn’t already exist, and you want to handle errors explicitly.
• Use INSERT IGNORE when you want to insert a row only if it doesn’t already exist, and you don’t care about handling errors.
• Use REPLACE when you want to ensure that a row is inserted or updated if it already exists, and you want to handle errors explicitly.

<?php
// Prepare an SQL statement to insert a new user into the 'users' table
$stmt = $conn->prepare("INSERT INTO users (name, email) VALUES (?, ?)");
// Bind the parameters to the SQL statement, specifying the types of the variables
$stmt->bind_param("ss", $name, $email);
// Assign values to the variables
$name = 'Jane Doe';
$email = 'jane.doe@example.com';
// Execute the prepared statement
$stmt->execute();
// Close the prepared statement
$stmt->close();
?>

1. Can I create a table without defining a primary key?

CREATE TABLE users (
  name VARCHAR(255),
  email VARCHAR(255)
);

2. How do I insert multiple rows in one query?

INSERT INTO users (name, email) VALUES ('John Doe', 'john.doe@example.com'), ('Jane Doe', 'jane.doe@example.com');

3. What’s the difference between CHAR and VARCHAR in MySQL?

CREATE TABLE users (
  name CHAR(10),
  email VARCHAR(255)
);

4. How do I copy and create a new table in MySQL?

CREATE TABLE new_users SELECT * FROM users;

5. How to create a database in MySQL?

CREATE DATABASE mydatabase;

• How to Create a New User and Grant Permissions in MySQL
• How to Use Indexes in MySQL
• How to Fix Corrupted Tables in MySQL
• How to Create and Manage Tables in SQL

• Finetuning tool:
  • MosaicML LLM Foundry
• Tasks:
  • Full pretraining from scractch
  • Finetuning of pretrained model
• Models:
  • Pretraining
    • MPT-125M, 350M, 760M, 1B, 3B, 7B, 13B, 30B, 70B
  • Finetuning
    • MPT-7B-Dolly-SFT
    • MPT-30B-Instruct
• Data:
  • Pretraining
    • C4
  • Finetuning
    • Dolly HH-RLHF
    • Instruct-v3
• Key metrics:
  • Token throughput (tok/s)
  • Model FLOPS uilization (MFU)
  • Runtime (wallclock)
• Scenario dimensions:
  • Pretraining
    • Model sizes: 125M, 350M, 760M, 1B
  • Finetuning
    • Number of nodes: 1, 2, 4, 8

Full pretraining

• In all cases, inference using the trained model, and model accuracy via evaluation on unseen testing data are verified.
• Model conversion after training to Hugging Face format, which gives a lighter weight model that is more efficient for inference, is also verified.
• The larger models in the MPT series, MPT-3B, -7B, -13B, -30B, and -70B, would run in the exact same way, but would require increasingly more wallclock time.
• Projection of runtime indicates that the largest model, MPT-70B, could be fully pretrained on DO in approximately 2 months on 8 nodes, or approximately 1 week on 64 nodes.
• Model FLOPS utilization, a measure of GPU usage more effective than raw utilization provided by LLM Foundry, is good, showing that the increased runtimes are genuine larger compute required by the larger models, not inefficiency of the infrastructure.

Finetuning

• The ideal speedup versus one node is N times for N nodes. We observe 2.67, 4.28, and 5.77x for 2, 4, and 8 nodes respectively. This indicates that we strongly gain from 2-4 nodes, a bit less so for 8. There is some overhead for model checkpoint saving after each epoch, but this is a normal operation.
• In all cases, inference using the trained model, and model accuracy via evaluation on unseen testing data are verified.
• Evaluation accuracy of the model on unseen data is higher than for the pretrained smaller models, as expected.

Datasets for pretraining and finetuning

Pretraining dataset

Finetuning dataset

MPT-7B-Dolly-SFT

MPT-30B-Instruct

Network Speed: NCCL Tests

mpirun \
-H hostfile \
-np 128 \
-N 8 \
--allow-run-as-root \
-x NCCL_IB_PCI_RELAXED_ORDERING=1 \
-x NCCL_IB_CUDA_SUPPORT=1 \
-x NCCL_IB_HCA^=mlx5_1,mlx5_2,mlx5_7,mlx5_8 \
-x NCCL_CROSS_NIC=0 -x NCCL_IB_GID_INDEX=1 \
$(pwd)/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Hardware and Software

DigitalOcean Bare-Metal machines

MosaicML LLM Foundry

• Emphasis on end-to-end usage, including data preparation, model training/finetuning, inference, and evaluation, like in real customer use cases.
• Command-line based interface that allows pretraining and finetuning to be instantiated in this setting without the requirement to write a lot of code specialized to a particular application.
• Shared disk multinode setup that avoided the need for setting up and usage of SLURM, unlike most multinode tools that introduce this complex additional overhead taken from the HPC world.
• Proof of efficient usage of GPU compute via the model FLOPS utilization (MFU) metric
• Existing benchmarks from MosaicML enabling calibration of expectations (e.g., of MFU), albeit on different machines so not 1:1 comparable with our numbers
• Support of real-world large language models in the MPT series
• Weights & Biases logging integration

• Anthropic’s HH-RLHF dataset
• Bare metal clusters
• C4 dataset
• Databricks’ dolly-15k dataset
• DigitalOcean AI/ML
• Dolly HH-RHLF dataset
• GPU usage
• Hugging Face
• Instruct-v3 dataset with correction
• LLM Foundry
• MosaicML MPT models
• RDMA over converged Ethernet (RoCE)

• Confidently write and run Python scripts while managing library installations.
• Practical experience with scikit-learn for training and evaluating machine learning models such as SVC and GradientBoostingClassifier.
• Understanding the distinction between model parameters and hyperparameters, training/validation/test sets, and evaluation metrics like accuracy, F1-score, and AUC.
• A Python environment where numpy, pandas, matplotlib, and scikit-learn libraries are installed.
• Comfort with concepts like bias-variance trade-off, loss functions, and gradient-based optimization.

Establish a Baseline Model

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

# Load dataset and split into train/validation sets
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Train a baseline SVM model with default hyperparameters
model = SVC(kernel='rbf', probability=True, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_val)

# Evaluate baseline performance
print("Baseline accuracy:", accuracy_score(y_val, y_pred))
print("Baseline F1-score:", f1_score(y_val, y_pred))
print("Baseline AUC:", roc_auc_score(y_val, model.predict_proba(X_val)[:, 1]))

Baseline accuracy: 0.9298245614035088
Baseline F1-score: 0.9459459459459459
Baseline AUC: 0.9695767195767195

Choose Hyperparameters to Tune

• C (Regularization parameter): Controls the trade-off between model complexity and error on training data.
  
  • Low C: The model applies stronger regularization, which can cause higher bias but reduce variance, increasing the risk of underfitting.
  • High C: Weaker regularization, allowing the model to fit the training data more closely. This reduces bias but increases variance, increasing the risk of overfitting.
• Gamma (Kernel coefficient): This determines how much influence a single training example has on the decision boundary.
  
  • Low gamma: Points have broader influence, resulting in smoother, more generalized decision boundaries.
  • High gamma: Each point impacts only its immediate neighborhood, which can lead to complex boundaries.

Tune Hyperparameters One-by-One (Manual Trial and Error)

• Start your experiments by choosing a single hyperparameter to adjust (such as C in SVM models).
• Choose a series of candidate values for testing based either on intuition or default settings. Hyperparameters often show non-linear effects, which makes it beneficial to test C values on a logarithmic scale(e.g., 0.01, 0.1, 1, 10, 100).
• For each selected value, train the model and evaluate its performance on the validation set.
• Plot or print the results to determine the best value before choosing it as your starting point.

for C in [0.01, 0.1, 1, 10, 100]:
    model = SVC(kernel='rbf', C=C, probability=True, random_state=42)
    model.fit(X_train, y_train)
    val_ac = accuracy_score(y_val, model.predict(X_val))
    print(f"C = {C:<5} | Validation Accuracy = {val_ac:.3f}")

C = 0.01  | Validation Accuracy = 0.842
C = 0.1   | Validation Accuracy = 0.912
C = 1     | Validation Accuracy = 0.930
C = 10    | Validation Accuracy = 0.930
C = 100   | Validation Accuracy = 0.947

for gamma in [1e-4, 1e-3, 1e-2, 0.1, 1]:
    model = SVC(kernel='rbf', C=1, gamma=gamma, probability=True, random_state=42)
    model.fit(X_train, y_train)
    val_ac = accuracy_score(y_val, model.predict(X_val))
    print(f"gamma = {gamma:<6} | Validation Accuracy = {val_ac:.3f}")

gamma = 0.0001 | Validation Accuracy = 0.930
gamma = 0.001  | Validation Accuracy = 0.895
gamma = 0.01   | Validation Accuracy = 0.640
gamma = 0.1    | Validation Accuracy = 0.632
gamma = 1      | Validation Accuracy = 0.632

• Manual one-at-a-time tuning helps you to build intuition about hyperparameters’ effects.
• Use the validation set for hyperparameter selection and keep the test set for final evaluation to avoid overfitting.
• Using multiple evaluation metrics(Such as F1 and AUC) for robust evaluation can help to identify overfitting and bias/variance issues at early stages.

Manual Grid Search (Systematic Exploration)

from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

param_grid = {
    "C":     [0.1, 1, 10, 50],
    "gamma": [1e-4, 1e-3, 0.01, 0.1]
}

best_ac = 0.0
best_f1  = 0.0
best_auc = 0.0
best_params = {}

for C in param_grid["C"]:
    for gamma in param_grid["gamma"]:
        model = SVC(kernel='rbf',
                    C=C,
                    gamma=gamma,
                    probability=True,
                    random_state=42)
        model.fit(X_train, y_train)

        # Predictions and probabilities
        y_v_pred  = model.predict(X_val)
        y_v_proba = model.predict_proba(X_val)[:, 1]

        # metrics computation
        ac = accuracy_score(y_val, y_v_pred)
        f1  = f1_score(y_val, y_v_pred)
        auc = roc_auc_score(y_val, y_v_proba)

        # You can Track best by accuracy or change to f1/auc as needed
        if ac > best_ac:
            best_ac    = ac
            best_f1     = f1
            best_auc    = auc
            best_params = {"C": C, "gamma": gamma}

        print(f"C={C:<4}  gamma={gamma:<6}  => "
              f"Accuracy={ac:.3f}  F1={f1:.3f}  AUC={auc:.3f}")

print(
    "\nBest combo:", best_params,
    f"with Accuracy={best_ac:.3f}, F1={best_f1:.3f}, AUC={best_auc:.3f}"
)

Best combo: {'C': 1, 'gamma': 0.0001} with Accuracy=0.930, F1=0.944, AUC=0.958

• The granularity of the grid matters. A coarse grid (few values) may overlook the best possible solution. A fine grid with numerous values increases the possibility of finding the best combination, but requires more training runs.
• When dealing with a large grid, it’s beneficial to apply cross-validation for a robust evaluation of each combination, but be aware that it increases computational requirements.
• Grid search suffers from the curse of dimensionality: Increasing hyperparameters or values can explode the number of combinations. At this point, random search emerges as a valuable strategy.

Manual Random Search for Hyperparameter Optimization

import random
import numpy as np
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

def random_search_svm(X_train, y_train, X_val, y_val, ntrials=10):
    """
    Run random search across C and gamma parameters for an RBF SVM model.
    Monitor optimal hyperparameters for peak Accuracy, F1-score, and ROC AUC values.
    """
    best = {
        'accuracy': {'score': 0, 'params': {}},
        'f1':       {'score': 0, 'params': {}},
        'auc':      {'score': 0, 'params': {}}
    }

    for i in range(1, ntrials + 1):
        # Log-uniform sampling
        C = 10 ** random.uniform(-1, 2)       # 0.1 to 100
        gamma = 10 ** random.uniform(-5, -2)  # 1e-5 to 1e-2

        # model training
        model = SVC(kernel='rbf', C=C, gamma=gamma,
                    probability=True, random_state=42)
        model.fit(X_train, y_train)

        # Prediction and evaluation
        y_pred = model.predict(X_val)
        y_proba = model.predict_proba(X_val)[:, 1]
        ac = accuracy_score(y_val, y_pred)
        f1 = f1_score(y_val, y_pred)
        auc = roc_auc_score(y_val, y_proba)

        # Print trial results
        print(f"Trial {i}: C={C:.4f}, gamma={gamma:.5f} | "
              f"Acc={ac:.3f}, F1={f1:.3f}, AUC={auc:.3f}")

        # For each metric, we will update the best
        if ac > best['accuracy']['score']:
            best['accuracy'].update({'score': ac, 'params': {'C': C, 'gamma': gamma}})
        if f1 > best['f1']['score']:
            best['f1'].update({'score': f1, 'params': {'C': C, 'gamma': gamma}})
        if auc > best['auc']['score']:
            best['auc'].update({'score': auc, 'params': {'C': C, 'gamma': gamma}})

    # For each metric, print summary of best hyperparameters
    print("\nBest hyperparameters by metric:")
    for metric, info in best.items():
        params = info['params']
        score = info['score']
        print(f"- {metric.capitalize()}: Score={score:.3f}, Params= C={params.get('C'):.4f}, gamma={params.get('gamma'):.5f}")

Best hyperparameters by metric:
- Accuracy: Score=0.939, Params= C=67.2419, gamma=0.00007
- F1: Score=0.951, Params= C=59.5889, gamma=0.00002
- Auc: Score=0.987, Params= C=59.5889, gamma=0.00002

Evaluate Model Performance and Select the Best Model

• Accuracy‐optimized: C ≈ 67.24, γ ≈ 7 × 10⁻⁵ (Accuracy = 0.9390)
• F1‐optimized: C ≈ 59.59, γ ≈ 2 × 10⁻⁵ (F1 = 0.9510)
• AUC‐optimized: C ≈ 59.59, γ ≈ 2 × 10⁻⁵ (AUC = 0.9870)

• If the ultimate goal is overall correctness, choose the Accuracy-optimized model.
  • Go for the F1-optimized model if balanced precision and recall matter.
  • The AUC-optimized model should be selected if ranking performance across thresholds is your priority.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score
import numpy as np

# 1. data loading and spliting into train+val vs. test
X, y = load_breast_cancer(return_X_y=True)
X_tem, X_test, y_tem, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 2.Spliting X_tem into training and validation sets. We want to reproduce the previous split
X_train, X_val, y_train, y_val = train_test_split(
    X_tem, y_tem, test_size=0.25, random_state=42, stratify=y_tem
)
# Note that 0.25 of 80% give us 20% validation, that matches our original 80/20 split

# 3. We merge train and validation data for our final testing
X_merged = np.vstack([X_train, X_val])
y_merged = np.hstack([y_train, y_val])

# 4. Retrain using our best hyperparameters
best_C = 59.5889      # replace with your chosen C
best_gamma = 2e-05    # replace with your chosen gamma

f_model = SVC(
    kernel='rbf',
    C=best_C,
    gamma=best_gamma,
    probability=True,
    random_state=42
)
f_model.fit(X_merged, y_merged)

# 5. Evaluate on hold-out test set
y_test_pred = f_model.predict(X_test)
y_test_proba = f_model.predict_proba(X_test)[:, 1]

test_ac = accuracy_score(y_test, y_test_pred)
test_f1 = f1_score(y_test, y_test_pred)
test_auc = roc_auc_score(y_test, y_test_proba)

print("Final model test accuracy:   ", test_ac)
print("Final model test F1-score:    ", test_f1)
print("Final model test ROC AUC:     ", test_auc)

from sklearn.ensemble import GradientBoostingClassifier

for lr in [0.001, 0.01, 0.1, 0.3, 0.7, 1.0]:
    model = GradientBoostingClassifier(n_estimators=50, learning_rate=lr, random_state=42)
    model.fit(X_train, y_train)
    val_acc = accuracy_score(y_val, model.predict(X_val))
    print(f"Learning rate {lr:.3f} => Validation Accuracy = {val_acc:.3f}")

Learning rate 0.001 => Validation Accuracy = 0.632
Learning rate 0.010 => Validation Accuracy = 0.939
Learning rate 0.100 => Validation Accuracy = 0.947
Learning rate 0.300 => Validation Accuracy = 0.956
Learning rate 0.700 => Validation Accuracy = 0.956
Learning rate 1.000 => Validation Accuracy = 0.965

• In regression/SVM, the C value or the regularization penalty (L1/L2 strength) represents a hyperparameter.
• Neural networks use dropout rate and L2 weight decay as regularization hyperparameters.
• Decision trees utilize max_depth or min_samples_leaf to reduce complexity through their regularization function.

• When using neural networks, the learning rate is usually the primary hyperparameter to tune. Subsequently, you can tune the number of layers/units, batch size, and regularization strength, which includes L2 and dropout parameters.
• With tree-based models, tree depth, the number of trees (estimators), and the learning rate (for boosted trees) are important.
• For the SVM algorithm, the parameters C and kernel gamma, when using the RBF kernel.
• For k-Nearest Neighbors, the number of neighbors is k.

• Hyperparameter Optimization with Keras Tuner
• JSON for Fine-tuning Machine Learning Models
• Deep Learning Metrics: Precision, Recall & Accuracy
• metrics-precision-recall-accuracy
• What Is Fine-Tuning in Machine Learning?

• Automated Machine Learning
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• Sequential model-based optimization in Python

In-Context Editing

LoRA-MoE Hybrid Tuning

VLM-Guided Noise Selection

Step 1 : Set up a GPU Droplet

Begin by setting up a DigitalOcean GPU Droplet, select AI/ML and choose the NVIDIA H100 option.

Step 2: Web Console

Once your GPU Droplet finishes loading, you’ll be able to open up the Web Console.web console

Step 3: Install Dependencies and Clone the ICEdit Repo

apt install python3-pip python3.10-venv
git clone https://github.com/River-Zhang/ICEdit

Step 4: Navigate to the correct repository

cd ICEdit

Step 5: Install Requirements

pip3 install -r requirements.txt
pip3 install -U huggingface_hub

Step 6: Obtain Flux Model Access

Agree to the conditions for the FLUX model since it will be used in the gradio implementation.

Step 7: Obtain Hugging Face Access Token

The HuggingFace token can be obtained from the Hugging Face Access Token page. Note that you may need to create a Hugging Face account. Ensure the appropriate boxes are selected before generating your token.

Step 8: Login to Hugging Face

huggingface-cli login

Step 9: Launch the Gradio app

python3 scripts/gradio_demo.py --share

There will be a share link which you can access from your browser.