{"id":3996,"date":"2020-08-18T00:54:53","date_gmt":"2020-08-18T00:54:53","guid":{"rendered":"https:\/\/mribeirodantas.xyz\/blog\/?p=3996"},"modified":"2025-01-31T04:37:50","modified_gmt":"2025-01-31T02:37:50","slug":"continuous-machine-learning-part-ii","status":"publish","type":"post","link":"https:\/\/mribeirodantas.xyz\/blog\/index.php\/2020\/08\/18\/continuous-machine-learning-part-ii\/","title":{"rendered":"Continuous Machine Learning – Part II"},"content":{"rendered":"Reading time: <\/span> 3<\/span> minutes<\/b><\/span><\/span>\n

This is a 3-part series about Continuous Machine Learning. You can check Part I here<\/a> and Part III here. This post is a continuation of the previous one, in which we initiated our experience on automating Data Science in GitHub with CML<\/span><\/a>. We will basically make use of Docker to improve the computation time in our GitHub Actions checks.<\/p>\n\n\n\n

You can think of a Docker image as taking a snapshot of the software environment of a project, and then being able to setup that snapshot on any other computer. When GitHub Actions is called, it loads your Docker image in their infrastructure and then runs your code. That’s why it’s quicker, because when you use a Docker container with your dependencies already installed, you don\u2019t have to spend time setting them up all over again on your GitHub Actions runner every time it is triggered, which is the way we did in the first part of this series<\/a>.<\/p>\n\n\n\n

Creating a Docker image<\/h4>\n\n\n\n
\"\"
Image from \u201cBuild a Docker Image just like how you would configure a VM<\/a>\u201d.<\/figcaption><\/figure>\n\n\n\n\n\n\n\n

As you can see on the image above, you write a text file called Dockerfile and the Docker app will create a Docker image based on the instructions contained in the Dockerfile. Then, with the Docker engine running in some infrastructure (GitHub’s infra, in our case), this image will be converted into a real container, like in the image below (There are three containers)..<\/p>\n\n\n\n

\"\"
Image from \u201cSi Docker m\u2019\u00e9tait cont\u00e9\u2026 2\u00e8me partie : plong\u00e9e \u00e0 l\u2019int\u00e9rieur des conteneurs<\/a>\u201d.<\/figcaption><\/figure>\n\n\n\n

We will base our Dockerfile on CML<\/span>’s official Dockerfile and it will look like the code below.<\/p>\n\n\n

\n\t\t\t\t
FROM dvcorg\/CML<\/span>\r\nRUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \\\r\n    add-apt-repository 'deb https:\/\/cloud.r-project.org\/bin\/linux\/ubuntu bionic-cran35\/' && \\  \r\n    apt update && \\\r\n    apt install -y r-base && \\\r\n    R --silent -e "install.packages(c(\\"igraph\\", \\"ppcor\\", \\"scales\\", \\"Rcpp\\"))" && \\\r\n    wget -c https:\/\/github.com\/miicTeam\/miic_R_package\/archive\/v1.4.2.tar.gz && \\\r\n    tar -xvzf v1.4.2.tar.gz && \\\r\n    R CMD INSTALL miic_R_package-1.4.2\/. --preclean<\/pre>\n\t\t\t<\/div>\n\n\n\n
As you can see, it has a lot of the code that we used to have in our CML<\/span>.yaml<\/code> file (in the first post of this series). That’s the magic! Because of that, our CML<\/span>.yaml<\/code> file is much shorter and should look like this:<\/p>\n\n\n
\n\t\t\t\t
name: dvc-CML<\/span>-miic\r\non: [push]\r\njobs:\r\n  run:\r\n    runs-on: [ubuntu-latest]\r\n    container: docker:\/\/mribeirodantas\/cml-test:r\r\n    steps:\r\n      - uses: actions\/checkout@v2\r\n\r\n      - name: cml_run\r\n        env:\r\n          repo_token: ${{ secrets.GITHUB_TOKEN }}\r\n          GDRIVE_CREDENTIALS_DATA: ${{ secrets.GDRIVE_CREDENTIALS_DATA }}\r\n        run: |\r\n          R --version\r\n\r\n          dvc pull\r\n          Rscript infer_network.R\r\n\r\n          # Write your CML<\/span> report\r\n          echo "## Model Metrics" > report.md\r\n          cat metrics.txt >> report.md\r\n          echo "## Data visualization" >> report.md\r\n          CML<\/span>-publish network_diagram.png --md >> report.md\r\n          CML<\/span>-send-comment report.md<\/pre>\n\t\t\t<\/div>\n\n\n\n
You can see the difference between the old and new CML<\/span>.yaml<\/code> in the image below (move the slider in the center to either side to see). You can also view it directly on GitHub clicking here<\/a>.<\/p>\n\n\n\n
\"\"\"\"<\/div><\/figure>\n\n\n\n
You can save the Dockerfile showed here in a file named Dockerfile in your repository (that right now should look like this<\/a>) and update your CML<\/span>.yaml<\/code> like the code above.<\/p>\n\n\n\n
To build it and publish it in Docker Hub (Think of it like GitHub but for Docker images), you gotta type the following Docker commands. You need Docker installed for that (instructions for Ubuntu here<\/a>). The first command takes a while, it’s creating your image and that’s the only time that it will really download and install everything. If you understood what I explained earlier, it should be clear that having a container pays off \ud83d\ude42<\/p>\n\n\n
\n\t\t\t\t
# Create the Docker image from the Dockerfile\r\nsudo docker build -t mribeirodantas\/CML<\/span>:r -f .\/Dockerfile .\r\n# Login to Docke Hub\r\nsudo docker login\r\n# Upload it to Docker Hub. GitHub will always download the\r\n# Docker image from there.\r\nsudo docker push mribeirodantas\/CML<\/span>:r<\/pre>\n\t\t\t<\/div>\n\n\n\n
Testing Docker image<\/h4>\n\n\n\n
We are not sure if this will work so we should create a branch and check what happens. Instead of creating pull requests from GitHub’s web interface, we will use the official GitHub command line gh<\/code>, just like we did in the first part of this series<\/a>.<\/p>\n\n\n
\n\t\t\t\t
git checkout -b docker_ghactions\r\ngit add -A\r\ngit commit -m 'Makes use of Docker to speed up GH Actions checks'\r\ngit push origin docker_ghactions\r\ngh pr create --title 'Makes use of Docker to speed up GH Actions checks'<\/pre>\n\t\t\t<\/div>\n\n\n\n
In the first part of this series, you probably remember that the checks took around 7 minutes to finish. In this branch that uses a Docker container it took less than 2 minutes. Taking into consideration this is a very simple project example and other projects could have much more dependencies, the benefits can be even larger than the ones we see here. Great, right? Let’s merge this push request.<\/p>\n\n\n
\n\t\t\t\t
# This command will merge all open push requests.\r\n# We only have one, so it will merge the one we have opened.\r\ngh pr merge<\/pre>\n\t\t\t<\/div>\n\n\n\n
One question that you may have is why leaving dvc pull<\/code> in the CML<\/span>.yaml<\/code> file instead of doing it only once in the Docker. After all, downloading big datasets every time should take a while right? Well, you are right, it may take a while. However, if nothing changed, dvc pull<\/code> quickly will sa yso and nothing will be done. And in case you changed your raw datasets, for example, it will download them.<\/p>\n\n\n\n
That’s it for today folks \ud83d\ude42<\/p>\n