Wednesday, October 27, 2021

[SOLVED] Integrating SonarQube within AWS CodePipeline: Connection Refused

Issue

tl;dr

CodePipeline crashes on the mvn sonar:sonar line of my buildspec.yml file with the following log (I formatted it a bit for better readability):

[ERROR] SonarQube server [http://localhost:9000] can not be reached 
...
[ERROR] Failed to execute goal 
        org.sonarsource.scanner.maven:sonar-maven-plugin:3.7.0.1746:sonar 
        (default-cli) on project myproject: 
        Unable to execute SonarQube: 
        Fail to get bootstrap index from server: 
        Failed to connect to localhost/127.0.0.1:9000: 
        Connection refused (Connection refused) -> [Help 1]

Goal

This is my first project with AWS, so sorry if I'm providing irrelevant informations.

I'm trying to deploy my backend API so that it's reachable by the public. Among other things, I want a CI/CD set up to automatically run tests and abort on failure or if a certain quality gate isn't passed. If everything went fine, then the new version should automatically be deployed online.

Current state

My pipeline automatically aborts when one of the tests fails, but that is about all I've gotten to properly do.

I've yet to figure out how to deploy (even manually) the API to be able to send requests to it. Maybe it's already done and I just don't know which URL to use, though.

Anyways, as it is, the CodePipeline crashes on the mvn sonar:sonar line of my buildspec.yml file.

The files

Here is my buildspec.yml:

version: 0.2 

phases: 
  install: 
    runtime-versions: 
      java: openjdk8 
    commands: 
      ############################################################################################## 
      ##### "cd / && ls" returns: [bin, boot, codebuild, dev, etc, go, home, lib, lib32, lib64, 
      #####                        media, mnt, opt, proc, root, run, sbin, srv, sys, tmp, usr, var] 
      ##### Initial directory where this starts is $CODEBUILD_SRC_DIR 
      ##### That variable contains something like "/codebuild/output/src511423169/src"
      ############################################################################################## 
      # Upgrade AWS CLI to the latest version 
      - pip install --upgrade awscli 
      # Folder organization 
      - cd /root 
      - codeAnalysisFolder="Sonar" # todo: refactor to include "/root" 
      - mkdir $codeAnalysisFolder && cd $codeAnalysisFolder 
      # Get SonarQube 
      - wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.1.0.31237.zip 
      - unzip ./sonarqube-8.1.0.31237.zip 
      # Launch SonarQube server locally 
      - cd ./sonarqube-8.1.0.31237/bin/linux-x86-64 
      - sh ./sonar.sh start 
      # Get SonarScanner 
      - cd /root/$codeAnalysisFolder 
      - wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.2.0.1873-linux.zip 
      - unzip ./sonar-scanner-cli-4.2.0.1873-linux.zip 
      - export PATH=$PATH:/root/$codeAnalysisFolder/sonar-scanner-cli-4.2.0.1873-linux.zip/bin/ # todo: .zip ?! 
  pre_build: 
    commands: 
      - cd $CODEBUILD_SRC_DIR 
      - mvn clean compile test 
      - mvn sonar:sonar 
  build: 
    commands: 
      - mvn war:exploded 
  post_build: 
    commands: 
      - cp -r .ebextensions/ target/ROOT/ 
      - aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template-file template-export.yml 
      # Do not remove this statement. This command is required for AWS CodeStar projects. 
      # Update the AWS Partition, AWS Region, account ID and project ID in the project ARN on template-configuration.json file so AWS CloudFormation can tag project resources. 
      - sed -i.bak 's/\$PARTITION\$/'${PARTITION}'/g;s/\$AWS_REGION\$/'${AWS_REGION}'/g;s/\$ACCOUNT_ID\$/'${ACCOUNT_ID}'/g;s/\$PROJECT_ID\$/'${PROJECT_ID}'/g' template-configuration.json 
artifacts: 
  type: zip 
  files: 
    - 'template-export.yml' 
    - 'template-configuration.json'

Here are the last few lines of the log of the failed build:

[INFO] User cache: /root/.sonar/cache 
[ERROR] SonarQube server [http://localhost:9000] can not be reached 
[INFO] ------------------------------------------------------------------------ 
[INFO] BUILD FAILURE 
[INFO] ------------------------------------------------------------------------ 
[INFO] Total time:  6.071 s 
[INFO] Finished at: 2019-12-18T21:27:23Z 
[INFO] ------------------------------------------------------------------------ 
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.7.0.1746:sonar (default-cli) on project myproject: Unable to execute SonarQube: Fail to get bootstrap index from server: Failed to connect to localhost/127.0.0.1:9000: Connection refused (Connection refused) -> [Help 1] 
[ERROR]  
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 
[ERROR] Re-run Maven using the -X switch to enable full debug logging. 
[ERROR]  
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException 

[Container] 2019/12/18 21:27:23 Command did not exit successfully mvn sonar:sonar exit status 1 
[Container] 2019/12/18 21:27:23 Phase complete: PRE_BUILD State: FAILED 
[Container] 2019/12/18 21:27:23 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: mvn sonar:sonar. Reason: exit status 1

And because you might also be interested in knowing the build's log related to the sh ./sonar.sh start command:

[Container] 2019/12/18 21:25:49 Running command sh ./sonar.sh start 
Starting SonarQube... 
Started SonarQube.

Additionally, here is my sonar-project.properties file:

# SONAR SCANNER CONFIGS 
sonar.projectKey=bullhubs 
# SOURCES 
sonar.java.source=8 
sonar.sources=src/main/java 
sonar.java.binaries=target/classes 
sonar.sourceEncoding=UTF-8 
# EXCLUSIONS 
# (exclusion of Lombok-generated stuff comes from the `lombok.config` file) 
sonar.coverage.exclusions=**/*Exception.java
# TESTS 
sonar.coverage.jacoco.xmlReportPaths=target/site/jacoco/jacoco.xml 
sonar.junit.reportsPath=target/surefire-reports/TEST-*.xml 
sonar.tests=src/test/java

The environment

(Sorry for the hidden infos: not being sure what should remain private, I went on the safe side. If you need any specific information, please let me know!)

I have an Elastic Beanstalk set up with the following properties:

EB instance

I also have an EC2 instance up and running:

EC2 instance

I also use a VPC.

What I've tried

I tried adding a bunch of entries into the inbound rules of my EC2's Security Group:

Security Group

I started from 0.0.0.0/0 : 9000, to then try 127.0.0.1/32 : 9000, to finally try All traffic. None of it worked, so the problem seems to be somewhere else.

I also tried changing some properties of the sonar-project.properties file, namely sonar.web.host and sonar.host.url, to try to redirect where the SonarQube server is hosted (I thought maybe I was supposed to point it to the EC2's IPv4 Public IP address or its attached Public DNS (IPv4)), but somehow the failing build log keeps displaying the failure to connect on localhost:9000 when trying to contact the SonarQube server.


Solution

I've figured it out.

Somehow, SonarQube reports having started properly despite that not being true. Thus, when you see this log after having ran your sh ./sonar.sh start command:

[Container] 2019/12/18 21:25:49 Running command sh ./sonar.sh start 
Starting SonarQube... 
Started SonarQube.

It isn't necessarily true that SonarQube's local server has successfully started. One would have to go into the logs folder of the SonarQube installation folder and read the sonar.log file to figure out that something was actually wrong and that the server was stopped...

In my case, it reported an error that JDK11 was required to run the server. To solve that, I changed the java: openjdk8 line of my buildspec.yml to java: openjdk11.

Then, I had to figure out that now a new log file was available to be read: es.log. When printing that file in the console, it was revealed to me that the latest ElasticSearch version (which is used by the latest SonarQube server version) does not allow itself to be ran by a root user. Thus, I had to create a new user group and edit some configuration file to run the server with that user:

  # Set up non-root user to run SonarQube
  - groupadd sonar
  - useradd -c "Sonar System User" -d $sonarPath/$sonarQube -g sonar -s /bin/bash sonar
  - chown -R sonar:sonar $sonarPath/$sonarQube  # recursively changing the folder's ownership
  # Launch SonarQube server locally
  - cd ./$sonarQube/bin/linux-x86-64
  - sed -i 's/#RUN_AS_USER=/RUN_AS_USER=sonar/g' sonar.sh  # enabling user execution of server
  - sh ./sonar.sh start

Complete solution

This gives us the following working version of buildspec.yml :

version: 0.2

phases:
  install:
    runtime-versions:
      java: openjdk11
    commands:
      ##############################################################################################
      ##### "cd / && ls" returns: [bin, boot, codebuild, dev, etc, go, home, lib, lib32, lib64,
      #####                        media, mnt, opt, proc, root, run, sbin, srv, sys, tmp, usr, var]
      ##### Initial directory where this starts is $CODEBUILD_SRC_DIR
      ##### That variable contains something like "/codebuild/output/src511423169/src"
      ##### This folder contains the whole structure of the CodeCommit repository. This means that
      ##### the actual Java classes are accessed through "cd src" from there, for example.
      ##############################################################################################
      # Upgrade AWS CLI to the latest version
      - pip install --upgrade awscli
      # Folder organization
      - preSonarPath="/opt/"
      - codeAnalysisFolder="Sonar"
      - sonarPath="$preSonarPath$codeAnalysisFolder"
      - cd $preSonarPath && mkdir $codeAnalysisFolder
      # Get SonarQube
      - cd $sonarPath
      - sonarQube="sonarqube-8.1.0.31237"
      - wget https://binaries.sonarsource.com/Distribution/sonarqube/$sonarQube.zip
      - unzip ./$sonarQube.zip
      # Set up non-root user to run SonarQube
      - groupadd sonar
      - useradd -c "Sonar System User" -d $sonarPath/$sonarQube -g sonar -s /bin/bash sonar
      - chown -R sonar:sonar $sonarPath/$sonarQube  # recursively changing the folder's ownership
      # Launch SonarQube server locally
      - cd ./$sonarQube/bin/linux-x86-64
      - sed -i 's/#RUN_AS_USER=/RUN_AS_USER=sonar/g' sonar.sh  # enabling user execution of server
      - sh ./sonar.sh start
      # Get SonarScanner and add to PATH
      - sonarScanner="sonar-scanner-cli-4.2.0.1873-linux"
      - cd $sonarPath
      - wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/$sonarScanner.zip
      - unzip ./$sonarScanner.zip
      - export PATH=$PATH:$sonarPath/$sonarScanner.zip/bin/ # todo: .zip ?!
  pre_build:
    commands:
      - cd $CODEBUILD_SRC_DIR
      - mvn clean compile test
#      - cd $sonarPath/$sonarQube/logs
#      - cat access.log
#      - cat es.log
#      - cat sonar.log
#      - cat web.log
#      - cd $CODEBUILD_SRC_DIR
      - mvn sonar:sonar
  build:
    commands:
      - mvn war:exploded
  post_build:
    commands:
      - cp -r .ebextensions/ target/ROOT/
      - aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template-file template-export.yml
      # Do not remove this statement. This command is required for AWS CodeStar projects.
      # Update the AWS Partition, AWS Region, account ID and project ID in the project ARN on template-configuration.json file so AWS CloudFormation can tag project resources.
      - sed -i.bak 's/\$PARTITION\$/'${PARTITION}'/g;s/\$AWS_REGION\$/'${AWS_REGION}'/g;s/\$ACCOUNT_ID\$/'${ACCOUNT_ID}'/g;s/\$PROJECT_ID\$/'${PROJECT_ID}'/g' template-configuration.json
artifacts:
  type: zip
  files:
    - 'template-export.yml'
    - 'template-configuration.json'

Cheers !



Answered By - payne
at