{"id":2952,"date":"2020-03-05T08:22:00","date_gmt":"2020-03-05T08:22:00","guid":{"rendered":"https:\/\/mribeirodantas.xyz\/blog\/?p=2952"},"modified":"2025-01-31T04:38:04","modified_gmt":"2025-01-31T02:38:04","slug":"r-dvc-and-rmarkdown","status":"publish","type":"post","link":"https:\/\/mribeirodantas.xyz\/blog\/index.php\/2020\/03\/05\/r-dvc-and-rmarkdown\/","title":{"rendered":"Manage your Data Science Project in R"},"content":{"rendered":"Reading time: <\/span> 9<\/span> minutes<\/b><\/span><\/span>\n

A simple project tutorial with R\/RMarkdown, Packrat, Git, and DVC.<\/strong><\/p>\n\n\n

\n
\"\"
Source: Here<\/a>.<\/figcaption><\/figure><\/div>\n\n\n

The pain of managing a Data Science project<\/h3>\n\n\n\n

Something has been bothering me for a while: Reproducibility and data tracking in data science projects. I have read about some technologies but had never really tried any of them out until recently when I couldn’t stand this feeling of losing track of my analyses anymore. At some point, I decided to give DVC a try after some friends, mostly Fl\u00e1vio Cl\u00e9sio<\/a>, suggested it to me<\/a>. In this post, I will talk about Git, DVC, R, RMarkdown, and Packrat, everything I think you may need to manage your Data Science project, but the focus is definitely on DVC.<\/p>\n\n\n\n\n\n\n\n

One of the things that I had been looking for is an approach that would allow me to track\/know the “version” of the dataset I used for a specific analysis when I wrote and run an R script or a set of R scripts. Have I run some-script.R<\/em> in the raw data or the preprocessed data? Have I run the some-other-script.R<\/em> in the dataset preprocessed this way or preprocessed that other way? What version of my dataset was used in the analysis done by the code committed put-a-git-commit-hash<\/em> here? In a few words: How to track code and data at the same time? (And of course, reports in RMarkdown being included is a plus).<\/p>\n\n\n\n

This tutorial will guide you through the management\/execution of a basic example of a Data Science project written in R. It’s not very close to what I do in my daily life and I assume it’s not what you do either, but it’s basic, simple and still useful for understanding how to use these tools together, mostly DVC.<\/p>\n\n\n\n

One second to talk about Git, Packrat and RMarkdown<\/h3>\n\n\n
\n
\"\"
Source: here<\/a>.<\/figcaption><\/figure><\/div>\n\n\n

I don’t want to get into much detail about git and packrat here (or how to install Git<\/a> and Packrat<\/a>), this is not the focus of the post. In a few words, git is a version control software which means that it helps you to track the evolution of your source code. Whenever you do some reasonable change to your code you take a snapshot of it (save point, whatever) through what is called a commit. This way, you can “time travel” to your project. You can find out when a bug was accidentally introduced in the code or work at the same time with several friends in the same code. Git will manage this for you. DVC comes to assist Git by adding this management\/tracking support to data files and to manage pipelines. There will be some Git commands here and there, and for the sake of not making this tutorial longer than it already is, I will not comment on them. You can get help here<\/a>.<\/p>\n\n\n

\n
\"\"
Source: here<\/a>.<\/figcaption><\/figure><\/div>\n\n\n

Packrat is a package dependency management for R which means that you can, for example, have a Data Science project using an R package version 1.X and using the same package in a different version 1.Z in another DS project, being both projects on the same machine. Besides, you can take snapshots with packrat too, making it easy to share your whole environment with colleagues or have it on another machine. If you’re from a Python background, this is like pip and virtualenv. Isolation and package dependency management.<\/p>\n\n\n

\n
\"\"
Source: here<\/a>.<\/figcaption><\/figure><\/div>\n\n\n

As its official website says, “Your data tells a story. Tell it with R Markdown. Turn your analyses into high quality documents, reports, presentations and dashboards”<\/em>. Think of it as Markdown + R technologies, turning it into markup language on steroids. <\/p>\n\n\n\n

DVC<\/h3>\n\n\n
\n
\"\"
Source: here<\/a>.<\/figcaption><\/figure><\/div>\n\n\n

When it comes to data files, the idea behind DVC is to create small text files (metadata) about the real data files and give them to git to take care, instead of the data files. Git is not made for big blobs or data files and by versioning your GB (TB?) datasets, you can make your repository slow to work with (and I won’t even mention cloning..). DVC will take care of the real files, while git will take care of metadata generated by DVC. DVC operates like Git and uses basically the same syntax: you add (dvc add<\/a>) and push (dvc push<\/a>) files to a remote location (remote storage, in dvc nomenclature). If you’re new to version control, this can take a little time to get used to. If you added a dataset to DVC, but you forgot to push and by accident you removed your dataset or lost the git repository, the remote storage (where DVC saves your real files) won’t have the dataset, just like the git remote repository wouldn’t have your source code, because you did not git push<\/em>. Everything is fine if you push, of course. However, in the same way that you have a “copy” of your source code in the remote repository, but also in your local repository, the real data files are both in the remote storage and in your local repository (in DVC cache hidden folder). Some common tasks in DVC can be automated to be integrated into your git commands. If you like this idea, dvc install<\/a> is what you’re looking for.<\/p>\n\n\n\n

If you have pip installed, you can easily install dvc with pip install dvc<\/em>. If you don’t, don’t worry. There are other ways to install it<\/a>, such as snap (GNU\/Linux) or brew (macOS).<\/p>\n\n\n\n

Let’s start our experiment.<\/h3>\n\n\n\n

The first thing we should do is to (1) create our Git repository, (2) start packrat and (3) DVC.<\/p>\n\n\n\n