feat: first implementation of the evaluation job (CM-1167)

[ulemons]

Summary

Introduces the skeleton of the projects_evaluation_worker, the second phase of the auto-onboarding pipeline. The worker picks up projects marked action = 'evaluate' in the projectCatalog table, runs an AI-based evaluation on each one, and updates the action to onboard or unsure along with evaluatedAt. The evaluator itself is a stub — the actual AI algorithm integration is tracked separately.

Changes

• New Temporal worker projects_evaluation_worker with workflow, activities, schedule (every Monday 02:00 UTC), Dockerfile and docker-compose YAML
• Added findProjectCatalogPendingEvaluation to the DAL — filters by action = 'evaluate', ordered by lfCriticalityScore DESC then createdAt ASC (highest score first, FIFO within same score)
• Evaluator stub in src/evaluator/evaluator.ts throws explicitly until the algorithm is wired

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

JIRA ticket

https://linuxfoundation.atlassian.net/browse/CM-1167

---

Note

Medium Risk
Introduces a new Temporal-scheduled worker that reads and updates projectCatalog rows, so misconfiguration or the current stubbed evaluator could incorrectly mark projects as unsure at scale. Changes are otherwise additive (new service, new DAL query) with limited impact on existing flows.

Overview
Adds an initial projects_evaluation_worker Temporal worker that runs on a weekly schedule, fetches projectCatalog entries with action = 'evaluate', evaluates them, and persists the resulting action plus evaluatedAt.

Includes new Docker build/compose setup for the worker and a new DAL helper findProjectCatalogPendingEvaluation that selects evaluable projects ordered by highest lfCriticalityScore then oldest createdAt.

^{Reviewed by Cursor Bugbot for commit 0de405a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

Adds the skeleton of a new projects_evaluation_worker Temporal service — the second phase of the auto-onboarding pipeline. The worker is scheduled weekly (Monday 02:00 UTC), pulls projectCatalog rows with action = 'evaluate' ordered by lfCriticalityScore, runs an AI evaluator (currently a stub that throws), and writes back action = 'onboard' | 'unsure' plus evaluatedAt. Includes the supporting DAL query, Dockerfile, docker-compose service definition, and pnpm lockfile entries.

Changes:

• New projects_evaluation_worker Temporal worker (workflow + activities + schedule + main + types + stub evaluator).
• New DAL helper findProjectCatalogPendingEvaluation ordering by lfCriticalityScore DESC NULLS LAST, createdAt ASC.
• Build/runtime plumbing: package.json, tsconfig, Dockerfile, docker-compose YAML, pnpm-lock, plus an unrelated tweak in scripts/cli.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/project-catalog/projectCatalog.ts	Adds findProjectCatalogPendingEvaluation DAL query.
services/apps/projects_evaluation_worker/tsconfig.json	Standard tsconfig extending the base.
services/apps/projects_evaluation_worker/src/workflows/evaluateProjects.ts	Main workflow: fetch a batch, iterate, evaluate each, count success/fail.
services/apps/projects_evaluation_worker/src/workflows.ts	Re-exports evaluateProjects.
services/apps/projects_evaluation_worker/src/schedules/scheduleProjectsEvaluation.ts	Registers weekly Monday 02:00 UTC Temporal schedule (SKIP overlap, 6h timeout).
services/apps/projects_evaluation_worker/src/main.ts	Bootstraps ServiceWorker, registers schedule on startup.
services/apps/projects_evaluation_worker/src/evaluator/types.ts	Defines IEvaluationInput, EvaluationOutcome, IEvaluationResult.
services/apps/projects_evaluation_worker/src/evaluator/evaluator.ts	Stub that throws until the AI algorithm is wired in.
services/apps/projects_evaluation_worker/src/activities/activities.ts	fetchPendingProjects + evaluateAndUpdateProject activities.
services/apps/projects_evaluation_worker/src/activities.ts	Re-exports activities.
services/apps/projects_evaluation_worker/package.json	Workspace package manifest for the new worker.
scripts/services/projects-evaluation-worker.yaml	docker-compose definitions (prod + dev) for the worker.
scripts/services/docker/Dockerfile.projects_evaluation_worker	Build/runtime Dockerfile for the worker.
scripts/cli	Un-comments IGNORED_SERVICES in clean-start-dev; unrelated to the PR's stated purpose.
pnpm-lock.yaml	Lockfile entries for the new workspace package.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

Comments suppressed due to low confidence (1)

services/apps/projects_evaluation_worker/src/schedules/scheduleProjectsEvaluation.ts:39

• throw new Error(err) wraps the original error object in a new Error whose message becomes "[object Object]" (or similar) and discards the original stack trace and properties. Prefer rethrowing the original error (throw err) or using new Error(err.message, { cause: err }). Note: this anti-pattern already exists in many other schedules in the codebase, but it would be cleaner not to propagate it into the new worker.

      throw new Error(err)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

Comments suppressed due to low confidence (3)

services/apps/projects_evaluation_worker/src/schedules/scheduleProjectsEvaluation.ts:26

• The workflowExecutionTimeout is set to 6 hours and the comment computes 100 × 3min ≈ 5h worst case, but this ignores activity retries: with maximumAttempts: 2 and a 3-minute startToCloseTimeout, the worst case is 100 × (3min × 2) = 10 hours plus retry backoff, well beyond the 6h ceiling. If many projects start failing/timing out, the workflow itself will be terminated mid-batch. Either increase the timeout to reflect retries, run evaluations in parallel, or drop the activity retry to 1.

        args: [{ batchSize: 100 }],
        // 100 projects × ~3min each = ~5h worst case; set ceiling with margin.
        workflowExecutionTimeout: '6 hours',

services/apps/projects_evaluation_worker/src/workflows/evaluateProjects.ts:50

• The workflow catches and swallows every per-project failure (lines 43-49) and the parent workflow never throws. As a result the workflow retry policy defined in scheduleProjectsEvaluation.ts (initialInterval/backoffCoefficient/maximumAttempts: 3) is effectively dead code — the only way the workflow can fail is if fetchPendingProjects exhausts its 3 internal retries. Either let some classes of failure bubble up so the schedule-level retry has meaning, or drop the workflow-level retry to avoid misleading configuration.

    try {
      await evaluateActivities.evaluateAndUpdateProject(project)
      succeeded++
    } catch (err) {
      // Log and continue — a single failure should not abort the whole batch.
      failed++
      log.error(
        `Evaluation failed for project id=${project.id} repoUrl=${project.repoUrl}: ${String(err)}`,
      )
    }
  }

services/apps/projects_evaluation_worker/src/schedules/scheduleProjectsEvaluation.ts:39

• throw new Error(err) wraps the original error in a generic Error, which loses the original stack trace and (when err is itself an Error) coerces it via toString() producing messages like "Error: Error: ...". Prefer throw err or throw new Error(String(err), { cause: err }). Note: this same anti-pattern already exists elsewhere in the codebase (e.g. scheduleProjectsDiscovery.ts:38), so this is just a stylistic suggestion for the new code.

      throw new Error(err)

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 7 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 5 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 6 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0de405a. Configure here.}