Gitlab runner times out while running lint command

Issue

I have the following GitLab CI step:

lint:
  stage: frontend_check
  only:
    changes:
      - frontend/**/*
  script:
    - cd frontend/ngapp
    - npm run lint
    - npm run prettier

Running this on GitLab will time out after two hours.

The raw log:

[0KRunning with gitlab-runner 16.4.0 (6e766faf)[0;m
[0K  on runner-192.168.63.53 U-zL7nfx, system ID: s_351d454b0ffc[0;m
section_start:1702987831:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m[0;m
[0KUsing Docker executor with image node:21.4.0 ...[0;m
[0KAuthenticating with credentials from /root/.docker/config.json[0;m
[0KPulling docker image node:21.4.0 ...[0;m
[0KUsing docker image sha256:b866e35a0dc4df85e168524b368567023eb22b06fe16f2237094e937fcd24d96 for node:21.4.0 with digest node@sha256:52206db44f7bb76dca465a9fae016922b6878c39261c87c9b719ae4d892fecfd ...[0;m
section_end:1702987834:prepare_executor
[0Ksection_start:1702987834:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner-u-zl7nfx-project-425-concurrent-0 via gitlab-runner...
section_end:1702987835:prepare_script
[0Ksection_start:1702987835:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1mFetching changes with git depth set to 50...[0;m
Reinitialized existing Git repository in /builds/ivu/landshut/.git/
[32;1mChecking out b6b09e82 as detached HEAD (ref is master)...[0;m
Removing frontend/ngapp/

[32;1mSkipping Git submodules setup[0;m
section_end:1702987841:get_sources
[0Ksection_start:1702987841:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m[0;m
[32;1mDownloading artifacts for maven_compile (635434)...[0;m
Downloading artifacts from coordinator... ok      [0;m  host[0;m=gitlab.webvalto.hu id[0;m=635434 responseStatus[0;m=200 OK token[0;m=64_xgXce
[32;1mDownloading artifacts for install_dependencies (635435)...[0;m
Downloading artifacts from coordinator... ok      [0;m  host[0;m=gitlab.webvalto.hu id[0;m=635435 responseStatus[0;m=200 OK token[0;m=64_xgXce
section_end:1702987878:download_artifacts
[0Ksection_start:1702987878:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:b866e35a0dc4df85e168524b368567023eb22b06fe16f2237094e937fcd24d96 for node:21.4.0 with digest node@sha256:52206db44f7bb76dca465a9fae016922b6878c39261c87c9b719ae4d892fecfd ...[0;m
[32;1m$ cd frontend/ngapp[0;m
[32;1m$ npm run lint[0;m

> customer-webportal-fe@1.0.2 lint
> ng lint --fix

Node.js version v21.4.0 detected.
Odd numbered Node.js versions will not enter LTS status and should not be used for production. For more information, please see https://nodejs.org/en/about/previous-releases/.
section_end:1702995031:step_script
[0K[31;1mERROR: Job failed: execution took longer than 2h0m0s seconds
[0;m

Running the same command on my machine with the same Node version completes in 5 seconds.

Update: npm install is done in a spearate job, and the folder is passed as an artifact. Adding the npm install command just in case directly to this job, did not solve the issue either.

update 2: I have registered my local machine as a runner and tried to run the job on it.

I saw in the top that when the job arrived ad the ng lint command, that the node process ran for a short period, then disappeared (hopefully meaning that the command was done) then the whole runner just hang forever, with leaving my PC in idle status.

.eslintrc.json:

{
  "root": true,
  "ignorePatterns": ["projects/**/*"],
  "overrides": [
    {
      "files": ["*.ts"],
      "parserOptions": {
        "project": ["tsconfig.json", "e2e/tsconfig.json"],
        "createDefaultProgram": true
      },
      "extends": [
        "plugin:@angular-eslint/recommended",
        "plugin:@angular-eslint/template/process-inline-templates"
      ],
      "rules": {
        "@angular-eslint/component-selector": [
          "error",
          {
            "prefix": "app",
            "style": "kebab-case",
            "type": "element"
          }
        ],
        "@angular-eslint/directive-selector": [
          "error",
          {
            "prefix": "app",
            "style": "camelCase",
            "type": "attribute"
          }
        ]
      }
    },
    {
      "files": ["*.html"],
      "extends": ["plugin:@angular-eslint/template/recommended"],
      "rules": {}
    }
  ]
}

Solution

The full gitlab template I was working with was:

image: node:21.4.0

stages:
  - maven_init # basic maven initialization
  - frontend_init # create frontend node environment for build
  - frontend_check # static checks before build
  - frontend_build # build the frontend app
  - pages # create documentation
  - test-server-deploy # deploy to test environment
  - e2e # isolated stage to test the currently deployed version on the  test server.
  - sonarqube-check # SonarQube code analysis

maven_compile:
  image: maven:latest
  needs: []
  stage: maven_init
  script:
    - mvn clean install -Ptest
  artifacts:
    paths:
      - ./frontend/ngapp/
  except:
    variables:
      - $RUN_E2E == "true"
      - $DEPLOY_TEST == "true"

lint:
  stage: frontend_check
  script:
    - cd frontend/ngapp
    - echo "Checking Angular CLI version..."
    - ng v
    - npm ci
    - echo "Running lint without rules/plugins..."
    # Temporarily disable rules/plugins here
    - CI=true ng lint
    - echo "Linting process completed or failed."
  cache:
    key: ${CI_COMMIT_REF_SLUG}-node-modules
    paths:
      - node_modules/

As you can see, there is a step, which runs a maven install first. It basicly runs a copy command ( <goal>copy-resources</goal>), and copies the project from the frontend/src/main/resources/ngapp to fronted/ngapp folder (among other things to prepare the bundle for being deployed). We used it with no problems for a long time, but after upgrading to angular 17, for some reason, the problem happened.

Replacing cd frontend/ngapp with frontend/src/main/resources/ngapp (so, running the ng lint command at the original location of the project, not in the copied folder` fixed the issue.

Running the exaclty same job on the original folder, where the project exists runs successfully. We are not sure why (tried different caching strategies; we run the npm install in the new folder, etc.) but it fixed the issue, and it does not get stuck anymore.

Answered By - ForestG