Configure gitlab CI for SonarQube with python projects

This blogpost is inspired by https://dev.to/mmphego/how-i-configured-sonarqube-for-python-code-analysis-with-jenkins-and-docker-28fm, and while working on it, I found the information is very trivial for gitlab CI.

Step 1: Activate pylint rules in Quality Profile

In my situation, sonarqube server is ready. We use it for java projects already, and the first thing is to activate pylint rules. Go to Quality Profiles tab, at the python panel, you have to create a new Quality Profile and activate pylint rules manually. Otherwise sonarqube server will not analyze the pylint report you upload later.

Inherit it from “Sonar Way”. Then activate in rules from Pylint rules repository.

You should see as the following picture after creating Quality Profile

ref:

Pylint and QualityProfile

Step 2: test locally

You can test first locally and it’s more convenient. So let’s start uploading the report from local.

Get coverage report by

(venv) my-terminal: pytest --cov-branch --cov=app tests/ --cov-report xml:coverage.xml

Get pylint report by

(venv) my-terminal: pylint app/ tests/ -r n — msg-template=”{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}” | tee pylint.txt

Note there is a space after the second colon(:)

Uploading report to sonar requires using sonar-scanner. You have to install it and then set the path. In my ubuntu, sonar-scanner is set in the file .bash_aliases

alias sonar-scanner=”~/Documents/sonar-scanner-4.0.0.1744-linux/bin/sonar-scanner”

and then you can run

sonar-scanner -Dsonar.projectKey=YOURPROJECT -Dsonar.python.pylint.reportPath=pylint.txt -Dsonar.host.url=YOUR_SONARQUBE_URL. -Dsonar.login=YOUR_LOGIN -Dsonar.language=py -Dsonar.python.xunit.reportPath=nosetests.xml -Dsonar.python.coverage.reportPaths=coverage.xml -Dsonar.sources=. -Dsonar.coverage.exclusions=**__init__**,tests/**,*.py -Dsonar.exclusions=*.xml

If you see the report, congrats! If not, check the Quality Profiles in the project page you just created, and it perhaps does not use pylint, but uses the default soanr-python. The Quality File must use the new one that you created before and pylint rules are activated.

Step 3: gitlab CI

OK! now locally works. The CI pipeline looks like below. Note that the image you use must have sonar-scanner installed

sonar-develop:
  image: PYTHON_IMG_WITH_SONAR_SCANNER
  stage: sonar
  script:
  - pip3 install -r requirements.txt
  - pip3 install pytest pylint
  - pytest --cov-branch --cov=app tests/ --cov-report xml:coverage.xml
  - 'pylint --exit-zero app/ tests/ -r n --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" | tee pylint.txt'  - pylint app/ tests/ -r n --msg-template="{path}':'{line}':' [{msg_id}({symbol}), {obj}] {msg}" | tee pylint.txt'
  - sonar-scanner -Dsonar.python.xunit.reportPath=nosetests.xml -Dsonar.python.coverage.reportPaths=coverage.xml
    -- -Dsonar.sources=. -Dsonar.coverage.exclusions=**__init__**,tests/**,config.py,manage.py -Dsonar.exclusions=*.xml
    -- -Dsonar.python.pylint.reportPath=pylint.txt -Dsonar.language=py
    -- -Dsonar.host.url=$SONAR_URL -Dsonar.login=$SONAR_LOGIN
    -- -Dsonar.gitlab.project_id=${CI_PROJECT_NAMESPACE}/$CI_PROJECT_ID
    -- -Dsonar.projectName=${CI_PROJECT_NAMESPACE}-${CI_PROJECT_NAME}
    -- -Dsonar.projectKey=${CI_PROJECT_NAMESPACE}-${CI_PROJECT_NAME}
    -- -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHA -Dsonar.gitlab.ref_name=$CI_COMMIT_REF_NAME
    -- -Dsonar.gitlab.url=$CI_PROJECT_URL -Dsonar.gitlab.project_id=$CI_PROJECT_ID

  only:
    refs:
      - develop
      - merge_requests
  except:
    refs:
      - tags
  when: manual

The critical part here is colon after a space (: ) in pylint command, so this command needs to be wrapped with quotes. Also note that

pylint --exit-zero

So it properly exits and pipeline can then success.

If you are confused about what to include and what to exclude, please refer to the following document from SonarQube.

Narrowing the Focus

For example, my application is flask, so coverage should exclude anything like

Dsonar.coverage.exclusions=**__init__**,tests/**,config.py,manage.py

and pylint should include

pylint app/ tests/