Allow packages and tools to use .tar.xz format

[PaulStoffregen]

Describe the request

Please support .tar.xz compression for packages and tools.

xz compression saves about 40% compared to bz2, which can substantially reduce download time for user and bandwidth cost for hosting large package and tools files.

Describe the current behavior

Here is a test package index using .tar.xz.

https://www.pjrc.com/teensy/td_158-beta3-xztest/package_teensy_index.json

Install Teensy version 0.58.3 to test.

Arduino CLI version

0.29

Operating system

N/A

Operating system version

any

Additional context

No response

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the nightly build
• My request contains all necessary details

[facchinm]

If we add new formats, thus obsoleting for good the java ide, I'd love if we'd also consider adding support for zstd, which is crazy fast for extraction and also provides good compression.

[PaulStoffregen]

This may be a stupid question, but is just releasing IDE 1.8.20 completely out of the question?

[ubidefeo]

I had a conversation with @per1234 and @cmaglie

We decided that we'll do the following:

• support .tar.xz in Arduino CLI (likely from 0.30.x)

• support zstd in the same version of Arduino CLI

• IDE 2.x will use this Arduino CLI version when available and we'll release it

• Platform specs will be updated to reflect the changes in supported formats

• IDE 1.x will remain untouched

• Arduino will keep releasing its platform index with the old format

• We'll announce the new formats and let developers implement them

• We'll specify that they are only available as of the specific versions of Arduino CLI and IDE 2.x

• Legacy Teensyduino won't break IDE 1.x because everything is bundled

• @PaulStoffregen will have to keep the format for future Teensyduino releases (if any) to the old format

Everyone happy with it?
I'm pushing this one into the next Sprints set

Thank you for your patience.

PS: great input, @PaulStoffregen
I'm always down with saving on data transfers 🙌🏼

[PaulStoffregen]

IDE1 frozen is nice for me. Really not missing porting Java patches and publishing new installers for each IDE release!

[PaulStoffregen]

I had to move the test case URL and .tar.xz files. New location:

https://www.pjrc.com/teensy/td_158-beta3-xztest/package_teensy_index.json

As before, install Teensy version 0.58.3 to test.

[PaulStoffregen]

Quick followup, we're testing performance of BZ2, ZST, XZ for installing Teensy's package, now that IDE releases have this additional compression support. Conversation and several people's experience is at this forum thread: https://forum.pjrc.com/index.php?threads/should-we-use-bz2-or-zst-or-xz-compression-for-teensy-downloads-in-arduino-ide-2-2-x.73981/ TL;DR = zstd looks like the clear winner.