How I Solved Performance Issues in CI/CD Pipelines with Large Artifacts in Git Repositories

Introduction

In a DevOps world where CI/CD pipelines are the heartbeat of development, dealing with large binary files in Git can be challenging. We faced this exact problem with vendor artifacts, roughly 1GB in size, committed directly into our Git repositories. The result? Slow pipelines, increased build times, and a compromised development workflow. I decided to combat this issue by implementing Git Large File Storage (LFS) and optimizing our Jenkins pipeline. Here’s journey in detail.

Understanding the Problem

Impact of Large Files in Git on CI/CD

Git, while powerful, is not optimized for handling large binary files. When these are part of your repository, it becomes sluggish and severely affects CI/CD pipelines. Here were our challenges:

1. Slow Repository Clones: A fresh clone in Jenkins took a considerable amount of time.
2. Increased Build Times: The delay in cloning affected subsequent stages.
3. Storage Issues: High costs and storage concerns on both CI/CD agents and source control systems.

Why We Chose Git LFS

After evaluating multiple solutions, including using separate storage for artifacts and performing shallow clones, we settled on Git LFS for its:

• Ease of use and integration into existing Git workflows.
• Efficient handling of large files, storing them outside the Git tree.
• Seamless compatibility with popular CI/CD tools like Jenkins.

Setting Up Git LFS

Installation and Configuration

Git LFS is an extension for Git and can be easily installed:

sudo apt-get install git-lfs
git lfs install
Code language: JavaScript (javascript)

Once installed, it allows you to specify which files should be tracked:

git lfs track "*.jar"Code language: JavaScript (javascript)

Migrating Existing Files to LFS

Existing large files can be migrated to LFS, but it requires rewriting history:

git lfs migrate import --include="*.jar"
Code language: JavaScript (javascript)

Don’t forget that rewriting history affects all branches and clones, so make sure to communicate this change to your team.

Integrating Git LFS with Jenkins Pipeline

Pre-Requisites

Ensure that Git LFS is installed on the Jenkins agent machines. You can add a “Preparation” stage in your Jenkinsfile to verify or install Git LFS.

Jenkinsfile Example with Git LFS and Selective Checkout

Here’s a simple Jenkins pipeline that checks out a specific branch and pulls only LFS objects required for that branch.

pipeline {
    agent any

    stages {
        stage('Preparation') {
            steps {
                script {
                    // Verify or install Git LFS on the Jenkins agent
                    sh '''
                    if ! git-lfs --version; then
                        sudo apt-get install git-lfs
                        git lfs install
                    fi
                    '''
                }
            }
        }
        
        stage('Checkout') {
            steps {
                script {
                    // Checkout code
                    checkout scm
                    
                    // Checkout LFS objects for the specific branch
                    sh 'git lfs pull origin <branch-name>'
                }
            }
        }

        stage('Build') {
            steps {
                sh './build.sh'
            }
        }

        stage('Test') {
            steps {
                sh './test.sh'
            }
        }
        
        stage('Deploy') {
            steps {
                sh './deploy.sh'
            }
        }
    }
}
Code language: PHP (php)

In this example, the git lfs pull command in the ‘Checkout’ stage ensures that only the necessary LFS objects for the specified branch are pulled, optimizing bandwidth and storage.

Best Practices for Git LFS in CI/CD

1. Communication: Always inform your team about the migration to avoid disruptions.
2. Backup: Backup your repositories before undertaking any history rewrite operation.
3. Validation: Validate the setup in a staging environment before rolling it out to production.

Financial and Resource Planning

Though Git LFS is a robust solution, it’s not entirely free. Most Git hosting solutions charge for LFS storage and bandwidth, so ensure to factor these costs into your project budget.

Conclusion

Managing large files in Git can be a headache, especially when they impact your CI/CD pipelines. However, with Git LFS and a correctly configured Jenkins pipeline, we efficiently mitigated the problems associated with large artifacts. The development workflow is now streamlined, the pipelines are faster, and developers are happier. It’s a win-win situation for everyone involved.