Skip to content

Improve the performance of provisioning and running of device tests #28065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 191 commits into from
Mar 5, 2025

Conversation

mattleibow
Copy link
Member

Description of Change

This PR has a bunch of things here, but they all aim to make device tests faster and prevent machine starvation.

The main changes are:

  • Speed up built task builds on CI by not building the platform binaries
  • Fail faster with Android boot failures (if it hasn't booted in 5 minutes, it will not boot so just retry)
  • Split some (just Android now) jobs into a build job and multiple run jobs (Android has 1 APK, but 10 API Levels)
  • Split the device tests step into 2 steps so we can retry just the run, not the build and run
  • Fix the Android runs for Linux so we can run Android things on Linux and free up macOS machines for Apple
  • Merge in all the work in [housekeeping] Use MSBuild for provisioning Android SDK, Appium #25288
    • We can use less of the private provisionator and more of the existing public tools
    • Switch Appium and Android SDK to provision via MSBuild targets so we can put all of the tool versions, API Levels and SDK packages info into the Versions.props to centralize the source of truth

@mattleibow mattleibow dismissed rmarinho’s stale review February 28, 2025 12:54

It never happened before because this step ran only on some jobs. I now made it always redirect the logs and it appears that on devdiv the reg vale was not set, but azure it is. I just made this step override the values. Now we should get windows logs for all runs.

@mattleibow mattleibow requested a review from rmarinho February 28, 2025 12:55
@rmarinho rmarinho merged commit 4c742b9 into main Mar 5, 2025
125 of 128 checks passed
@rmarinho rmarinho deleted the dev/linux-android branch March 5, 2025 12:28
bhavanesh2001 pushed a commit to bhavanesh2001/maui that referenced this pull request Mar 7, 2025
…otnet#28065)

* Use MSBuild for provisioning

Appium and Android SDK to start

* We use Nexus 5X as the device type currently

* Fix provision call / logs path

* Ignore exit codes

NPM may not return an ok exit code if there's no appium installed.  Similarly, we may get no driver list versions and a failed exit code when we query for appium drivers if none are installed.

* Fix call to target

* Provision JDK as well

* Provision from pipeline

* yaml fixes

* Yaml fix

* Add project to build invocation

* Restore dotnet tools first

* Restore tools after clear cache

* Ensure winget is installed first

* Download jdk msi and install it

* Different download location

* Property name for downloaded file

* Separate provisioning steps

* Log more jdk list info

* Update provision.yml

* Fix batching

* Log msiexec install

* Split up android provisioning steps

* Fix indentation

* Update tool with bugfix

* JDK: Try default installdir on windows

* Android: Option to install only default api levels

* Update provision.yml

* Fix parameter passing

* MSBuild log diag

* Skip setting JAVA_HOME

* Compatible gradle versions for JDK 17

* Set JDK_HOME again

* Try to pick up jdk for gradle

* Fix type/var name

* Always look up android home

* Simplify locate of best JDK

* Cake runs with stdout

* Disable the cleaner as per comment

* Less verbose logging

* Update and comment out Tools entry point for now

* Remove dup merge step

* Add condition for skipping appium doctor

* Skip the unsupported task

* Only install the image and avd wea are using

* windows or mac/linux

* apples don't like linus

* this too

* including

* better linux things

* no kvm on azure

* Revert "no kvm on azure"

This reverts commit cdd69a3.

* AndroidSdkDirectory

* log the info

* info

* and latest

* saf

* Provision the JDK on linux too

* sudo

* list packages

* remove latest

* this

* this

* disable kvm for now

* Use new emu

* better things

* provisioning creates the emulators

* KVM

* order

* try this

* machines

* better logs

* oops

* cleanups

* test boot

* oops

* test passed

* try splitting

* this

* inside

* oops

* oooops

* 2

* no need for menus in the build

* -v:detailed

* this

* yaml amirite

* faster!

* moar!

* ${{ parameters.appArtifactName }}

* this

* try this

* that

* more

* ${{ parameters.platform }}

* obj

* oops

* cake

* attempt 10 million

* skipDotNet

* sighs

* whaa

* ne

* oops

* safdd

* suspision

* maybe this one?

* to pwsh or not to pwsh

* give up for now

* test-only

* log things

* that

* this

* where is it???

* this

* windows has 2 separate builds

* maybe this

* merge the files and jobs

* this

* oops

* always publish

* better

* we can skip .net if we don't need it

* names

* this

* try this

* try again

* )

* yaml is not yaml-ing

* debugging yaml

* eq

* seems to always fail the first time

* split to speed up retries

* this

* better

* try harder

* better booting and reties

* this does nothing

* faster as this works

* faster!

* enable all

* Update eng/devices/android.cake

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Rui doesn't like my yaml

* flip that

* this

* FASTER!

* FAAAAAAASTERRRRR!!!!

* too fast?

* 27 is just terrible

* force

* mac is bad sometimes

* can't be const

* that bad!

* params can't use expressions

* revert for now

* no idea anymore

* sigh

* this

* CLEANUP IS COOL

* always()

---------

Co-authored-by: redth <jondick@gmail.com>
Co-authored-by: Jonathan Dick <jodick@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
rmarinho pushed a commit that referenced this pull request Mar 11, 2025
…28065)

* Use MSBuild for provisioning

Appium and Android SDK to start

* We use Nexus 5X as the device type currently

* Fix provision call / logs path

* Ignore exit codes

NPM may not return an ok exit code if there's no appium installed.  Similarly, we may get no driver list versions and a failed exit code when we query for appium drivers if none are installed.

* Fix call to target

* Provision JDK as well

* Provision from pipeline

* yaml fixes

* Yaml fix

* Add project to build invocation

* Restore dotnet tools first

* Restore tools after clear cache

* Ensure winget is installed first

* Download jdk msi and install it

* Different download location

* Property name for downloaded file

* Separate provisioning steps

* Log more jdk list info

* Update provision.yml

* Fix batching

* Log msiexec install

* Split up android provisioning steps

* Fix indentation

* Update tool with bugfix

* JDK: Try default installdir on windows

* Android: Option to install only default api levels

* Update provision.yml

* Fix parameter passing

* MSBuild log diag

* Skip setting JAVA_HOME

* Compatible gradle versions for JDK 17

* Set JDK_HOME again

* Try to pick up jdk for gradle

* Fix type/var name

* Always look up android home

* Simplify locate of best JDK

* Cake runs with stdout

* Disable the cleaner as per comment

* Less verbose logging

* Update and comment out Tools entry point for now

* Remove dup merge step

* Add condition for skipping appium doctor

* Skip the unsupported task

* Only install the image and avd wea are using

* windows or mac/linux

* apples don't like linus

* this too

* including

* better linux things

* no kvm on azure

* Revert "no kvm on azure"

This reverts commit cdd69a3.

* AndroidSdkDirectory

* log the info

* info

* and latest

* saf

* Provision the JDK on linux too

* sudo

* list packages

* remove latest

* this

* this

* disable kvm for now

* Use new emu

* better things

* provisioning creates the emulators

* KVM

* order

* try this

* machines

* better logs

* oops

* cleanups

* test boot

* oops

* test passed

* try splitting

* this

* inside

* oops

* oooops

* 2

* no need for menus in the build

* -v:detailed

* this

* yaml amirite

* faster!

* moar!

* ${{ parameters.appArtifactName }}

* this

* try this

* that

* more

* ${{ parameters.platform }}

* obj

* oops

* cake

* attempt 10 million

* skipDotNet

* sighs

* whaa

* ne

* oops

* safdd

* suspision

* maybe this one?

* to pwsh or not to pwsh

* give up for now

* test-only

* log things

* that

* this

* where is it???

* this

* windows has 2 separate builds

* maybe this

* merge the files and jobs

* this

* oops

* always publish

* better

* we can skip .net if we don't need it

* names

* this

* try this

* try again

* )

* yaml is not yaml-ing

* debugging yaml

* eq

* seems to always fail the first time

* split to speed up retries

* this

* better

* try harder

* better booting and reties

* this does nothing

* faster as this works

* faster!

* enable all

* Update eng/devices/android.cake

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Rui doesn't like my yaml

* flip that

* this

* FASTER!

* FAAAAAAASTERRRRR!!!!

* too fast?

* 27 is just terrible

* force

* mac is bad sometimes

* can't be const

* that bad!

* params can't use expressions

* revert for now

* no idea anymore

* sigh

* this

* CLEANUP IS COOL

* always()

---------

Co-authored-by: redth <jondick@gmail.com>
Co-authored-by: Jonathan Dick <jodick@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@github-actions github-actions bot locked and limited conversation to collaborators Apr 5, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants