New AOT options for Karma also run code coverage on component HTML files

[hamfastgamgee]

Command

test

Is this a regression?

• Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was

No response

Description

When using the new-in-19.2.x option in angular.json to run Karma using AOT, I was surprised to find that my code coverage metrics suddenly got a lot worse. Looking into the output of "ng test" with code coverage enabled, I found that the number of statements and lines touched roughly doubled compared to when I run the same tests with AOT turned off. By looking at the files generated by karma-coverage, I found that in AOT mode the component HTML files are also being instrumented for code coverage now. I assume this is because AOT mode compiles the templates into the Javascript for test execution.

Was this intentional? If it was, at the very least it probably should have been mentioned in the release notes. I can't find any way to configure the codeCoverageExclude options to omit the component HTML files. I would imagine nearly every app will see a drop in their code coverage as a result of this difference, especially given that most of the IDE tooling around code coverage doesn't even consider the possibility that HTML files would be included. It's also not obvious at all how to cover the HTML; I have files that are listed as fully covered with basically no difference in the tests compared to ones that aren't marked as covered at all.

Minimal Reproduction

Create a simple component with a simple test and run Karma with code coverage enabled both with AOT enabled and AOT disabled. With AOT enabled, the component HTML will be instrumented in addition to the Typescript. With AOT disabled, only the Typescript is instrumented.

Exception or Error

Your Environment

_                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/
    

Angular CLI: 19.2.6
Node: 22.14.0
Package Manager: npm 11.2.0
OS: win32 x64

Angular: 19.2.5
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.1902.6
@angular-devkit/build-angular     19.2.6
@angular-devkit/core              19.2.6 (cli-only)
@angular-devkit/schematics        19.2.6
@angular/cdk                      19.2.8
@angular/cli                      19.2.6
@angular/material                 19.2.8
@angular/material-luxon-adapter   19.2.8
@schematics/angular               19.2.6
rxjs                              7.8.2
typescript                        5.8.2
zone.js                           0.15.0

Anything else relevant?

No response

[dgp1130]

Running code coverage on AOT templates is working as intended. If you have an @if (...) { ... } then it does make sense that users would want code coverage information about that. JIT physically can't support that requirement, so this is something that only works with AOT.

I believe sourcemaps should mostly make this work. However if code coverage information for templates appears inaccurate, then I would recommend filing a separate issue with a minimal reproduction so we can look into whatever might be causing it.

[hamfastgamgee]

I think that's a perfectly fine feature to have and I agree a desirable one. However, the fact remains that it was not documented anywhere and it is likely to obliterate code coverage thresholds in a lot of apps. (I know in our applications we do almost exclusive testing of the functions behind component events and not the triggering of the events themselves. Our thresholds dropped between 10-30% depending on the app.) I would like to suggest some combination of the following:

1. some notice in the release notes that this is going to happen when someone opts in to AOT mode,
2. separating the coverage out based upon Typescript vs. template (I realize this may be complicated or impossible with the tooling available), or
3. (my preference if there can only be one) that there be an option to do exclusions to prevent this code from being instrumented. You can't add *.html to codeCoverageExcludes[] because they are no longer HTML by the time the test executes.

I don't doubt that the output is accurate, although again, the IDE tooling does not consider that templates are even something that will be subjected to code coverage, so there's no real feedback other than guess and check.

In any case, thanks for your consideration. :)

[dgp1130]

I definitely see the frustration in trying to adopt AOT testing and suddenly seeing this (seemingly unrelated) drop in code coverage. Ultimately this is effectively a bug fix from a code coverage perspective, given that the JIT number is really just wrong, and the AOT number is a more accurate representation of your application's code coverage. A team choosing to turn this off or disabling it is really just intentionally misleading themselves about the state of their application.

1. some notice in the release notes that this is going to happen when someone opts in to AOT mode,

Release notes are fairly ephemeral and most people don't read them anyways. Mentioning this in the option's documentation is likely to draw more eyeballs long term.

2. separating the coverage out based upon Typescript vs. template (I realize this may be complicated or impossible with the tooling available),

I'm not sure how that would work, and even if it did, this would likely just create more confusion for developers since they would now have to manage two code coverage numbers.

3. that there be an option to do exclusions to prevent this code from being instrumented. You can't add *.html to codeCoverageExcludes[] because they are no longer HTML by the time the test executes.

I can agree it would be useful for any excludes to work with *.html. Unfortunately I don't think any test runners / code coverage tools actually work that way? By the time the runner is executing, the *.html file is JS, so this is really asking to exclude coverage based on sourcemap name, and I don't think that's a common feature in test runners.

[hamfastgamgee]

Yeah, that's fair, and thanks for the engagement/consideration. Put me down as voting for an improvement to the option documentation to assist the next developer, then. I suspect I'm not the only person who wouldn't immediately think of templates as "code" in the traditional sense, although clearly they are, especially in the AOT case. Definitely agree it's better to have tooling not lie to people.