Troubleshooting SSL – GitLab Documentation

• Useful OpenSSL debugging commands Common
• SSL

errors

• Git-LFS and other embedded services written in the golang report Custom certificate

signed by an unknown authority

• Reconfiguration failed due to

certificates

• Missing or skipped custom
• certificates Custom certificates not detected

Let’s Encrypt

• Certificate signed by an unknown authority Let’s
• Encrypt fails to reconfigure

Using an internal CA certificate with GitLab

• X.509 key value mismatch error

Using GitLab Runner with a GitLab instance configured

• with an internal CA

certificate or self-signed certificate

• Duplicate a remote GitLab repository that uses a

self-signed SSL certificate

• Unable to perform Git operations due to an internal or self-signed
• certificate SSL_connect incorrect Schannel version number
• : SEC_E_UNTRUSTED_ROOT

This page contains a list of common SSL-related errors and scenarios that you may encounter when working with GitLab. It should serve as an addition to the main SSL documentation:

• Omnibus SSL configuration

.

• Self-signed certificates or custom Certificate Authorities for GitLab Runner

.

• Configure HTTPS manually

.

Useful OpenSSL Debugging Commands

Sometimes it is useful to get a better picture of the SSL certificate chain by looking at it directly at the source. These commands are part of the standard library of OpenSSL tools for diagnostics and debugging.

• Make a test connection to the host over HTTPS. Replace HOSTNAME with your GitLab URL (excluding HTTPS) and replace the port with the port that serves HTTPS connections (usually 443):

  The echo command sends a null request to the server, causing it to close the connection instead of waiting for additional input. You can use the same command to test remote hosts (for example, a server hosting an external repository) by replacing HOSTNAME:port with the domain and port number of the remote host.

  The output of this command displays the certificate chain, any public certificates presented by the server, along with validation or connection errors if they occur. This makes it a quick check for any immediate issues with your SSL settings.

• View the details of a certificate in text form with x509. Be sure to replace

  /path/to/certificate.crt with the certificate path:

  For example, GitLab automatically retrieves and places certificates acquired from Let’s Encrypt in /etc/gitlab/ssl/hostname.crt. You can use the x509 command with that path to quickly display certificate information (for example, host name, issuer, validity period, and so on).

  If there is a problem with the certificate, an error occurs.

• Obtain a certificate from a server and decode it. This combines the previous two commands to get the SSL certificate from the server and decode it into text

:

Common SSL errors

SSL certificate issue

1. : Unable to obtain certificate from local issuer

   This error indicates that the client cannot obtain the root CA. To work around this issue, you can trust the root CA of the server you are trying to connect to on the client, or modify the certificate to present the full chained certificate on the server to which you are trying to connect.

2. Unable to verify first

   certificate

   This error indicates that the server is presenting an incomplete certificate chain. To correct this error, you will need to replace the server certificate with the full chained certificate. The complete order of the certificate chain must consist first of the server certificate, followed by all intermediate certificates, with the root CA at the end.

certificate

3. signed by an unknown authority

   This error indicates that the client does not trust the certificate or CA. To correct this error, the client connecting to the server must trust the certificate or CA.

4. SSL certificate issue: Self-signed certificate in certificate chain

   This error indicates that the client does not trust the certificate or CA. To correct this error, the client connecting to the server must trust the certificate or CA.

5. x509: Certificate is based on Legacy Common Name field, use

   SAN instead

   This error indicates that SANs (subjectAltName) must be configured on the certificate. For more information, see this issue.

Git-LFS and other integrated services written in golang report custom certificate signed by an unknown authority

The gitlab-workhorse and other services written in golang use the golang crypto/tls library instead of OpenSSL. Add the following entry in /etc/gitlab/gitlab.rb

to fix the problem as reported:

Reconfigure Fails Due to Certificates

Check /opt/gitlab/embedded/ssl/certs and

delete any files other than README.md that are not valid X.509 certificates

.

Custom certificates Missing or omitted

GitLab versions 8.9.0, 8.9.1, and 8.9.2 mistakenly used the /etc/gitlab/ssl/trusted-certs/ directory. It is safe to delete this directory if it is empty. If it still contains custom certificates, move them to /etc/gitlab/trusted-certs/ and run gitlab-ctl reconfigure.

If no symbolic links are created in /opt/gitlab/embedded/ssl/certs/ and you see the message “Skipping cert.pem” after running gitlab-ctl reconfigure, that means there may be one of four problems:

The file in /etc/

1. gitlab/trusted-certs/ is a symbolic link The file

is not a

2. valid PEM or DER-encoded

certificate

3. Perl is not installed on the operating system required to c_rehash to properly bind certificates
4. Certificate contains the TRUSTED string

Test the validity of the certificate

by using the following commands

:Invalid certificate files

produce the following output

:

To check if c_rehash is not binding the certificate due to the lack of a

perl interpreter:

If you see this message, you will need to install perl using your distribution’s package manager.

If you

inspect the certificate itself, look for the TRUSTED

string:If you do, as in the

previous example, try deleting the TRUSTED string and run

gitlab-ctl reconfigure again.

Custom certificates not detected

If after running gitlab-ctl reconfigure:

no symbolic links are created in /opt/gitlab/embedded/ssl/certs/; you have placed custom certificates in /
1. etc/gitlab/trusted-certs/; and
3. You don’t see any skipped or symbolically linked custom certificate messages You

might encounter an issue where Omnibus GitLab believes that custom certificates have already been added

.

To resolve this, delete the hashed trusted certificates directory:

Then run gitlab-ctl reconfigure again. The reconfiguration should now detect and symbolically link your custom certificates.

Let’s Encrypt Certificate signed by an unknown authority

The initial implementation of the Let’s Encrypt integration only

used the certificate, not the entire certificate chain.

Starting at 10.5.4, the full certificate chain will be used. For installations that already use a certificate, the change will not occur until the renewal logic indicates that the certificate is about to expire. To force it first, run the following

Where HOSTNAME is the host name of the certificate.

Let’s Encrypt fails to reconfigure

When you reconfigure, there are common scenarios where

Let’s Encrypt can fail: Let’s Encrypt may fail if your server cannot reach

Let’s Encrypt’s

1. verification servers

   or vice versa:

   If you’re having trouble reconfiguring GitLab due to Let’s Encrypt, make sure you have ports 80 and 443 open and accessible.

Your domain’s

2. Certificate Authority Authorization (CAA) record does not allow Let’s Encrypt to issue a certificate for your domain. Look for the following error in the reconfiguration output:

3. If you are using a test domain as a gitlab.example.com, without a certificate, you will see the error of not being able to request the certificate shown above. In that case, disable Let’s Encrypt by setting letsencrypt[‘enable’] = false in /etc/gitlab/gitlab.rb.

You can test your domain with the Let’s Debug diagnostic tool. It can help you figure out why you can’t issue a Let’s Encrypt certificate.

Using an internal CA certificate with GitLab After you configure a

GitLab instance with an

internal CA certificate, you might not be able to access it using various CLI tools. You may experience the following issues

:

• Curl

error:

• Testing using the rail console also fails

:

• The SSL certificate issue: Unable to get certificate from local issuer error is displayed when configuring a mirror from this GitLab instance.

• OpenSSL works by specifying the path to the certificate:

If you have the problems described above, add your certificate to /etc/gitlab/trusted-certs, and then run sudo gitlab-ctl reconfigure.

X.509

key values mismatch error

After you configure your instance with a certificate package, NGINX may display the following

error message:SSL: error:

0B080074:x509 certificate routines:X509_check_private_key:key values

do not match

This error message means that the server certificate and the key that you provided do not match. You can confirm this by running the following command and then comparing the output:

The following is an example of an md5 output between a matching key and a certificate. Note

the matching md5 hashes:

This is an opposite output with a mismatched key and certificate showing different md5 hashes:

If the two outputs differ as in the example above, there is a mismatch between the certificate and the key. Contact your SSL certificate vendor for further assistance.

Using GitLab Runner with

a GitLab instance configured

with an internal CA certificate or self-signed certificate

In addition to getting the errors mentioned in Using an internal CA certificate with GitLab, your CI pipelines can get stuck in the Pending state. In the executor logs you may see the following error message:

Follow the details in Self-signed certificates or Custom certification authorities for GitLab Runner.

Duplicate a remote GitLab repository that uses a self-signed SSL certificate When you configure a local GitLab instance

to mirror a repository on a remote GitLab instance that uses a self-signed certificate, you may see the SSL certificate issue: Self-signed certificate error message in the user interface.

The cause of the problem can be confirmed by checking whether

:

• curl fails

:

• Tests using the Rails console also fail

:

To resolve this issue:

Add the self-signed certificate from the remote GitLab instance to the /etc/gitlab

• /trusted-certs directory on the local GitLab instance, and then run sudo gitlab-ctl reconfigure according to the instructions for installing custom public certificates.

If your local GitLab instance

• was installed using Helm charts, you can add your self-signed certificate to your GitLab instance.

You may also receive another error message when you try to mirror a repository from a remote GitLab instance that uses

a self-signed certificate:

In this case, the issue may be related to the certificate itself:

1. Validate that your self-signed certificate is not missing a common name. if so regenerate a valid

certificate

2. Add the certificate to /etc/gitlab/trusted-certs

.

3. Run sudo gitlab-ctl reconfigure.

Unable

to perform Git operations due to an internal or self-signed certificate If

your GitLab instance is using a self-signed certificate, or if the certificate is signed by an internal certification authority (CA), you may experience the following errors when you try to perform Git operations

:

To resolve this issue:

• If possible, use SSH remotes for all Git operations. This is considered safer and more comfortable to use.
• If you must use HTTPS remote controls, you can try the following:

  • Copy the self-signed certificate or internal root CA certificate to a local directory (for example, ~/.ssl) and configure Git to trust your certificate:

  • Disable SSL verification in your Git client. This is intended to be a temporary measure, as it could be considered a security risk.

SSL_connect incorrect version number

Incorrect configuration can result

in: gitlab-rails/exceptions_json.log entries containing:

gitlab-workhorse/current

• contains: gitlab-rails

/sidekiq

• .log or sidekiq/current containing:

Some of these errors come from the Excon Ruby gem, and could be generated in circumstances where GitLab is configured to start an HTTPS session on a remote server that it only serves HTTP.

One scenario is that you are using object storage, which is not served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, but object storage responds with simple HTTP.

schannel: SEC_E_UNTRUSTED_ROOT

If you are on Windows and get the following error

:

You need to specify that Git should use OpenSSL:

Alternatively, you can ignore SSL verification by running: