A detailed example of data loaders with PyTorch

PHOTO EMBED

Fri Jul 02 2021 13:43:59 GMT+0000 (Coordinated Universal Time)

Saved by @Quinn #gnn #pytorch #trainset #loaddata

# Load entire dataset
X, y = torch.load('some_training_set_with_labels.pt')

# Train model
for epoch in range(max_epochs):
    for i in range(n_batches):
        # Local batches and labels
        local_X, local_y = X[i*n_batches:(i+1)*n_batches,], y[i*n_batches:(i+1)*n_batches,]

        # Your model
        [...]
content_copyCOPY

https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel