Getting started with DVC: purpose, how it works, and the commands

Datasets and model files do not belong in Git. A CSV of a few hundred megabytes, and a model larger than that, makes clone, diff, and rollback slow. DVC (Data Version Control) keeps those files outside Git, commits a small pointer instead, and uses the same idea for the training pipeline: when code, parameters, or data change, only the affected steps run again.
DVC does not replace Git. Git keeps the code and the pointers. DVC keeps the data, the models, and the pipeline outputs.
What it is for
Day to day, that is four jobs:
- Version data and models. Each
dvc addor pipeline rerun records a content hash that lines up with a Git commit. - Sync those large files between machines. One copy lives in the local cache, another on a remote (disk, NFS, S3, SSH, or WebDAV).
- Turn “prepare data → train → write metrics” into a pipeline. Steps whose dependencies did not change are skipped.
- Switch data and models together with the code. After
git checkout, rundvc checkout.
Git LFS can also replace a large file with a pointer, and the blob storage is usually tied to the Git host. A DVC remote can be object storage or an SSH machine you already operate, and the pipeline is part of the tool: it records which dataset, which script, and which parameters a training run depended on.
How it works
The workspace, the local cache, and the remote each hold a copy of the data. Git holds only the pointers.
flowchart LR workspace["Data and models in the workspace"] -->|dvc add / dvc repro| cache["Local cache .dvc/cache"] cache -->|dvc push| remote["Remote"] remote -->|dvc pull| cache cache -->|dvc checkout| workspace meta[".dvc, dvc.yaml, dvc.lock"] -->|git commit| gitrepo["Git repository"]
Pointer files
dvc add data/raw.csv does three things: hash the file contents, store the file in .dvc/cache, write data/raw.csv.dvc beside it, and add data/raw.csv to .gitignore.
A pointer file looks like this (the hash and size are filled in by the command):
1outs:
2- md5: 22a1a2931c8370d3aeedd7183606fd7f
3 size: 1048576
4 hash: md5
5 path: raw.csv
The file is only a few lines, so it can be committed. After someone clones the repo they still do not have the CSV. dvc pull fetches the bytes from the remote by that hash.
Directories work the same way. dvc add data/ tracks the whole directory. The cache stores each file plus a directory listing.
Content-addressed storage
The cache lives in .dvc/cache and is stored by MD5 of the file contents, not by file name. Identical contents are stored once. Change one byte and the hash changes, so the cache gains a new object. The old object stays until you clean it up. Checking out an old commit restores the file as long as that hash is still in the cache or on the remote.
Workspace files are often hard links into the cache, and they are read-only, so editing the workspace does not overwrite a historical version in the cache. To change a tracked file, run dvc unprotect, edit it, then dvc add again.
A pipeline records dependencies
A pipeline is dvc.yaml. Each stage has:
cmd: the command to rundeps: inputs, data files or codeparams: parameters read fromparams.yamlouts: data or models the command writes; DVC caches thesemetrics: a metrics file. Withcache: falseit stays out of the cache, and a small file can go straight into Git
dvc repro writes dvc.lock. The lockfile records the hash of every dependency and output from that run, and it should be committed. The next rerun compares the workspace with the hashes in the lock:
- Dependencies, parameters, and the command are unchanged: skip the stage
- An input to one stage changed: start at that stage and follow the graph downstream
- Before a stage runs, DVC deletes the
outsthat stage declared, then runscmd. The script has to create its output directories, and it should overwrite output files rather than append to them
dvc.yaml says what to do. dvc.lock says what the last run produced. Both go into Git. The CSV and model.pkl go into the cache and the remote.
If a metrics file sets cache: false, its hash is still written to dvc.lock and the file itself stays in the workspace. You can run dvc metrics diff and compare AUC across two commits without pulling the data.
Where configuration goes
dvc init creates .dvc/. Commit .dvc/config, which holds the remote URL. Write secrets to .dvc/config.local. That file is ignored by default.
One complete version is these four pieces:
| Where | What |
|---|---|
| Git | Code, params.yaml, dvc.yaml, dvc.lock, *.dvc, .dvc/config |
.dvc/cache | Data and models, stored by hash |
| Remote | A copy of the cache, so other machines can dvc pull |
.dvc/config.local | Credentials for the remote, not committed |
Commands
The commands sit next to Git: one side commits pointers, the other pushes data.
Init and remotes
1dvc init
2
3# Local disk or a shared mount. -d makes it the default remote
4dvc remote add -d storage /mnt/dvc-storage
5
6# Object storage. Install the plugin first, and keep credentials in the local config
7# pip install dvc-s3
8dvc remote add -d storage s3://team-ml/churn/dvc
9dvc remote modify --local storage access_key_id 'AKIA...'
10dvc remote modify --local storage secret_access_key '...'
11
12dvc remote list
13dvc remote default storage
An SSH remote is ssh://user@host/data/dvc-cache. Pick one remote for the team and commit its address in .dvc/config.
Synology WebDAV as DVC storage
When a Synology NAS is already on the home or office LAN, DSM’s WebDAV Server can be the DVC remote. Another machine then uses dvc pull to fetch the data and the model.
On the NAS:
- Install WebDAV Server from Package Center. Enable HTTPS and leave the port at the default
5006. On a LAN you can also enable HTTP on5005. - Control Panel → Shared Folder, and create a folder that holds only the cache, for example
dvc. Turn the recycle bin off for that folder, so cleaning up old versions actually returns disk space. - Control Panel → User & Group, and create a user
dvc. Grant that user read/write on the shared folder, and allow WebDAV Server in the user’s application permissions.
In the project, install the WebDAV extra, commit the address, and keep the account on the machine:
1pip install 'dvc[webdav]'
2
3dvc remote add -d nas-webdav webdavs://192.168.1.20:5006/dvc
4dvc remote modify --local nas-webdav user 'your-user'
5dvc remote modify --local nas-webdav password 'your-password'
Replace 192.168.1.20 with the NAS address on the LAN. dvc is the shared folder name. For HTTP only, use webdav://192.168.1.20:5005/dvc.
DSM ships with a self-signed certificate. If dvc push fails on certificate verification, turn verification off for LAN use:
1dvc remote modify nas-webdav ssl_verify false
If the NAS already has your own certificate, or you exported the CA, point ssl_verify at that pem file.
What you commit is .dvc/config:
1[core]
2 remote = nas-webdav
3['remote "nas-webdav"']
4 url = webdavs://192.168.1.20:5006/dvc
5 ssl_verify = false
The account stays in .dvc/config.local, which is already ignored:
1['remote "nas-webdav"']
2 user = your-user
3 password = your-password
Push and check it like any other remote:
1dvc push
2dvc status --cloud
If dvc push fails while listing directories, enable Depth Infinity in WebDAV Server and push again. A reverse proxy in front of port 443 has to allow uploads larger than a single model file. The built-in 5006 port has no such limit.
Track data
1dvc add data/raw.csv # write the .dvc file, cache the data, ignore the original
2dvc unprotect data/raw.csv # make a read-only hard link writable before editing
3dvc add data/raw.csv # refresh the hash after the edit
4dvc commit # copy workspace edits of tracked files back into the cache and update pointers
5dvc remove data/raw.csv.dvc # stop tracking the file and update the ignore rules
Use dvc add for new data, or to update a file tracked by its own .dvc file. Leave pipeline outputs alone. Change a dependency and run dvc repro.
Sync and switch versions
1dvc status # workspace compared with the pointers
2dvc status --cloud # what the local cache is missing relative to the remote
3dvc push # cache → remote
4dvc fetch # remote → cache, workspace unchanged
5dvc pull # fetch, then checkout
6dvc checkout # restore the workspace from the cache using the pointers in the current Git commit
Switching code and switching data are two commands:
1git checkout <commit>
2dvc checkout
If the cache does not contain that hash, dvc checkout reports it missing. Run dvc pull next.
Pipelines
Write dvc.yaml by hand, or generate it with dvc stage add. This is the same train stage as in the example below:
1dvc stage add -n train \
2 -d src/train.py -d data/train.csv -d data/test.csv \
3 -p train \
4 -o models/model.pkl \
5 -M metrics.json \
6 python src/train.py
-M declares a metrics file. The default is cache: false.
1dvc repro # rerun stages whose dependencies changed, in graph order
2dvc repro train # check train and its upstream stages only
3dvc repro --dry # print the commands that would run
4dvc dag # show the stage graph
5dvc metrics show # current metrics
6dvc metrics diff HEAD # compare workspace metrics with a commit
7dvc params diff HEAD # compare parameters
Experiments and cache cleanup
When you are still trying parameters and do not want a commit yet, the experiment commands record a run and leave the workspace as it was:
1dvc exp run -S train.max_depth=6
2dvc exp show
3dvc exp apply <experiment> # bring that run back into the workspace, then commit it yourself
The local cache grows with every version. dvc gc deletes nothing unless you pass a scope. -w keeps only the objects referenced by the pointers in the current workspace, and drops other versions from the local cache:
1dvc gc -w
Objects you already dvc pushed can be restored later with dvc fetch. --cloud deletes the remote with the same scope. Versions removed there are gone from that remote, so use it only after you are sure the old data can go.
A typical example
The dataset below is for churn prediction. DVC tracks the raw CSV. Split and train are two stages. On another machine, git clone and dvc pull retrieve the same data and the same model.
Repository layout:
1.
2├── data/
3│ └── raw.csv
4├── src/
5│ ├── prepare.py
6│ └── train.py
7├── params.yaml
8├── dvc.yaml
9└── dvc.lock # written by dvc repro
data/raw.csv needs feature columns and label. These three rows only show the shape. A real file needs enough rows of both labels, or one side of the split will be missing a class:
1tenure,monthly_charges,label
23,29.85,0
312,56.95,0
41,89.10,1
params.yaml:
1prepare:
2 test_size: 0.2
3 seed: 7
4train:
5 max_depth: 4
6 seed: 7
src/prepare.py reads the parameters and the raw data, and writes the two splits:
1from pathlib import Path
2
3import pandas as pd
4import yaml
5from sklearn.model_selection import train_test_split
6
7
8def main():
9 with open("params.yaml") as f:
10 params = yaml.safe_load(f)["prepare"]
11 df = pd.read_csv("data/raw.csv")
12 train, test = train_test_split(
13 df,
14 test_size=params["test_size"],
15 random_state=params["seed"],
16 )
17 Path("data").mkdir(exist_ok=True)
18 train.to_csv("data/train.csv", index=False)
19 test.to_csv("data/test.csv", index=False)
20
21
22if __name__ == "__main__":
23 main()
src/train.py reads the splits and writes the model and the metrics:
1import json
2from pathlib import Path
3
4import joblib
5import pandas as pd
6import yaml
7from sklearn.metrics import roc_auc_score
8from sklearn.tree import DecisionTreeClassifier
9
10
11def main():
12 with open("params.yaml") as f:
13 params = yaml.safe_load(f)["train"]
14 train = pd.read_csv("data/train.csv")
15 test = pd.read_csv("data/test.csv")
16 features = [col for col in train.columns if col != "label"]
17
18 model = DecisionTreeClassifier(
19 max_depth=params["max_depth"],
20 random_state=params["seed"],
21 )
22 model.fit(train[features], train["label"])
23 proba = model.predict_proba(test[features])[:, 1]
24
25 Path("models").mkdir(exist_ok=True)
26 joblib.dump(model, "models/model.pkl")
27 Path("metrics.json").write_text(
28 json.dumps({"roc_auc": float(roc_auc_score(test["label"], proba))}, indent=2)
29 + "\n"
30 )
31
32
33if __name__ == "__main__":
34 main()
dvc.yaml:
1stages:
2 prepare:
3 cmd: python src/prepare.py
4 deps:
5 - src/prepare.py
6 - data/raw.csv
7 params:
8 - prepare
9 outs:
10 - data/train.csv
11 - data/test.csv
12 train:
13 cmd: python src/train.py
14 deps:
15 - src/train.py
16 - data/train.csv
17 - data/test.csv
18 params:
19 - train
20 outs:
21 - models/model.pkl
22 metrics:
23 - metrics.json:
24 cache: false
Track the data the first time
1git init
2dvc init
3pip install dvc pandas scikit-learn pyyaml joblib
4
5dvc add data/raw.csv
6git add data/raw.csv.dvc data/.gitignore .dvc
7git commit -m "track raw dataset with dvc"
8
9dvc remote add -d storage /mnt/dvc-storage
10git add .dvc/config
11git commit -m "add dvc remote"
12dvc push
If the Synology NAS is not mounted as a share, use the WebDAV remote from the previous section instead of /mnt/dvc-storage. The git add and dvc push steps stay the same.
At this point Git has the pointer and the remote address, and the remote has the CSV. data/raw.csv does not show up as a file to commit in git status.
Run the pipeline and commit
After the scripts, params.yaml, and dvc.yaml are in place:
1dvc repro
2dvc metrics show
dvc repro runs prepare, then train, and writes dvc.lock. data/train.csv, data/test.csv, and models/model.pkl go into the cache. metrics.json stays in the workspace.
1git add dvc.yaml dvc.lock params.yaml src metrics.json
2git commit -m "add prepare and train stages"
3dvc push
4git push
Both git commit and dvc push have to happen. Pushing only Git tells others the hash, but they cannot fetch the file. Pushing only DVC leaves them without the hash to pull.
Another machine
1git clone git@github.com:org/churn.git
2cd churn
3dvc pull # fetch raw.csv, the splits, and the model from the lockfile
4dvc repro # nothing changed, so both stages are skipped
Change a parameter
Set train.max_depth in params.yaml from 4 to 8:
1dvc repro
2dvc metrics diff HEAD
3dvc params diff HEAD
prepare is unchanged, so it is skipped. train sees a new parameter and runs again. After you accept the metrics:
1git add dvc.lock params.yaml metrics.json
2git commit -m "train with max_depth 8"
3dvc push
When the data itself changes, edit data/raw.csv and run dvc add data/raw.csv again. Once the hash changes, dvc repro starts over from prepare.
To compare a few depths before committing:
1dvc exp run -S train.max_depth=6
2dvc exp show
Pick one run, bring it back with dvc exp apply, then commit and dvc push as above.
Return to the previous version
1git checkout HEAD~1
2dvc checkout
Code, parameters, and dvc.lock return to the previous commit, and the CSV and model in the workspace return to that hash. That is the version you get by tying DVC to one Git commit.
Command details are in the DVC Command Reference. Pipeline fields are in dvc.yaml. Get “pointers in Git, contents on the remote, reruns decided by the lockfile” working before you add experiments and cache cleanup.
Comments