Library discovery broken by use of __has_include() preprocessor operator

[KurtE]

Describe the problem

Suppose I have a library I am working on, in this case MTP_Teensy (in this case: https://github.com/KurtE/MTP_Teensy/tree/try_auto_detect_sd_media)

Where I would like to detect if the sketch is using the SD library and if so I define an override to a method.

In the library header file, I have:

#if defined(__has_include) && __has_include(<SD.h>)
#include <SD.h>
#endif

And if it is included:

  // Add a file system to the list of storages that will be seen by
  // the host computer.  Returns the index of the item within the list
  #if defined(__SD_H__)
  uint32_t addFilesystem(SDClass &disk, const char *diskname);
  #endif
  uint32_t addFilesystem(FS &disk, const char *diskname);

Now I have real simple sketch using it:

#include <SD.h>
#include <MTP_Teensy.h>

#define CS_SD BUILTIN_SDCARD  // Works on T_3.6 and T_4.1
//#define CS_SD 10  // Works on SPI with this CS pin
void setup()
{
  // mandatory to begin the MTP session.
  MTP.begin();

  // Add SD Card
  SD.begin(CS_SD);
  MTP.addFilesystem(SD, "SD Card");
}

void loop() {
  MTP.loop();  //This is mandatory to be placed in the loop code.
}

End of the compile output.

"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\tools\\teensy-tools\\1.56.53/teensy_size" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino-sketch-E4981CB39DB7E3541CDD1889059DFD07/Example_3_simple_SD.ino.elf"
Memory Usage on Teensy 4.1:
  FLASH: code:85572, data:11324, headers:8572   free for files:8020996
   RAM1: variables:13536, code:82888, padding:15416   free for local variables:412448
   RAM2: variables:28800  free for malloc/new:495488
Multiple libraries were found for "SD.h"
  Used: D:\GitHub\SD
  Not used: C:\arduino-1.8.19\libraries\SD
  Not used: C:\Users\kurte\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.56.53\libraries\SD
Multiple libraries were found for "SdFat.h"
  Used: D:\GitHub\SdFat
  Not used: C:\Users\kurte\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.56.53\libraries\SdFat
Using library SD at version 2.0.0 in folder: D:\GitHub\SD 
Using library SdFat at version 2.1.2 in folder: D:\GitHub\SdFat 
Using library SPI at version 1.0 in folder: C:\Users\kurte\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.56.53\libraries\SPI 
Using library MTP_Teensy at version 1.0.0 in folder: D:\GitHub\MTP_Teensy 

However if I swap the order of the two includes, that is:

#include <MTP_Teensy.h>
#include <SD.h>

#define CS_SD BUILTIN_SDCARD  // Works on T_3.6 and T_4.1
//#define CS_SD 10  // Works on SPI with this CS pin
void setup()
{
  // mandatory to begin the MTP session.
  MTP.begin();

  // Add SD Card
  SD.begin(CS_SD);
  MTP.addFilesystem(SD, "SD Card");
}

void loop() {
  MTP.loop();  //This is mandatory to be placed in the loop code.
}

This will not build. In fact SD.h will not loaded:

C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\tools\\teensy-tools\\1.56.53/precompile_helper" "C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\hardware\\avr\\1.56.53/cores/teensy4" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino-sketch-E4981CB39DB7E3541CDD1889059DFD07" "C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\tools\\teensy-compile\\1.56.1/arm/bin/arm-none-eabi-g++" -x c++-header -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -std=gnu++14 -fno-exceptions -fpermissive -fno-rtti -fno-threadsafe-statics -felide-constructors -Wno-error=narrowing -mthumb -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -D__IMXRT1062__ -DTEENSYDUINO=157 -DARDUINO=10607 -DARDUINO_TEENSY41 -DF_CPU=720000000 -DUSB_MTPDISK_SERIAL -DLAYOUT_US_ENGLISH "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\hardware\\avr\\1.56.53/cores/teensy4" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino-sketch-E4981CB39DB7E3541CDD1889059DFD07/pch/Arduino.h" -o "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino-sketch-E4981CB39DB7E3541CDD1889059DFD07/pch/Arduino.h.gch"
"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\tools\\teensy-compile\\1.56.1/arm/bin/arm-none-eabi-g++" -c -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -std=gnu++14 -fno-exceptions -fpermissive -fno-rtti -fno-threadsafe-statics -felide-constructors -Wno-error=narrowing -mthumb -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -D__IMXRT1062__ -DTEENSYDUINO=157 -DARDUINO=10607 -DARDUINO_TEENSY41 -DF_CPU=720000000 -DUSB_MTPDISK_SERIAL -DLAYOUT_US_ENGLISH "-IC:\\Users\\kurte\\AppData\\Local\\Temp\\arduino-sketch-E4981CB39DB7E3541CDD1889059DFD07/pch" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\hardware\\avr\\1.56.53\\cores\\teensy4" "-Ic:\\Users\\kurte\\Documents\\Arduino\\libraries\\MTP_Teensy\\src" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino-sketch-E4981CB39DB7E3541CDD1889059DFD07\\sketch\\Example_3_simple_SD.ino.cpp" -o "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino-sketch-E4981CB39DB7E3541CDD1889059DFD07\\sketch\\Example_3_simple_SD.ino.cpp.o"
C:\Users\kurte\AppData\Local\Temp\.arduinoIDE-unsaved2022523-40232-1qadghd.p0xs\Example_3_simple_SD\Example_3_simple_SD.ino: In function 'void setup()':
C:\Users\kurte\AppData\Local\Temp\.arduinoIDE-unsaved2022523-40232-1qadghd.p0xs\Example_3_simple_SD\Example_3_simple_SD.ino:12:3: error: 'SD' was not declared in this scope
   SD.begin(CS_SD);
   ^
C:\Users\kurte\AppData\Local\Temp\.arduinoIDE-unsaved2022523-40232-1qadghd.p0xs\Example_3_simple_SD\Example_3_simple_SD.ino:4:15: error: 'BUILTIN_SDCARD' was not declared in this scope
 #define CS_SD BUILTIN_SDCARD  // Works on T_3.6 and T_4.1
               ^
C:\Users\kurte\AppData\Local\Temp\.arduinoIDE-unsaved2022523-40232-1qadghd.p0xs\Example_3_simple_SD\Example_3_simple_SD.ino:12:12: note: in expansion of macro 'CS_SD'
   SD.begin(CS_SD);
            ^

Using library MTP_Teensy at version 1.0.0 in folder: D:\GitHub\MTP_Teensy 
exit status 1

Compilation error: 'SD' was not declared in this scope

The build output showed a few hints. When the build succeeded, the list of libraries at the end showed:

Using library SD at version 2.0.0 in folder: C:\Users\kurte\Documents\Arduino\libraries\SD 
Using library SdFat at version 2.1.2 in folder: C:\Users\kurte\Documents\Arduino\libraries\SdFat 
Using library SPI at version 1.0 in folder: C:\Users\kurte\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.56.53\libraries\SPI 
Using library MTP_Teensy at version 1.0.0 in folder: C:\Users\kurte\Documents\Arduino\libraries\MTP_Teensy

The failed version only only showed:
Using library MTP_Teensy at version 1.0.0 in folder: C:\Users\kurte\Documents\Arduino\libraries\MTP_Teensy
Also looking that the compile commands at the -I we see in the one that worked

"-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\hardware\\avr\\1.56.53\\cores\\teensy4"
"-IC:\\Users\\kurte\\Documents\\Arduino\\libraries\\SD\\src" 
"-IC:\\Users\\kurte\\Documents\\Arduino\\libraries\\SdFat\\src" 
"-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\hardware\\avr\\1.56.53\\libraries\\SPI"
"-IC:\\Users\\kurte\\Documents\\Arduino\\libraries\\MTP_Teensy\\src"

The one that failed:

"-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\hardware\\avr\\1.56.53\\cores\\teensy4" 
"-IC:\\Users\\kurte\\Documents\\Arduino\\libraries\\MTP_Teensy\\src"

So the search path did not find the SD.h file so the __has_include(<SD.h>)
was false... Also was not found in main sketch so we then get the compile errors

To reproduce

More or less covered above.

I have RC8 of the IDE installed.
I have the current Teensy boards installed: https://www.pjrc.com/teensy/td_156/package_teensy_index.json
I have the MTP_Teensy installed.

Expected behavior

Should build with includes in either order

Arduino CLI version

Arduino IDE2 RC8 - just downloaded

Operating system

Windows

Operating system version

Windows 10

Additional context

More comments up on the Teensy forum thread I opened yesterday:
https://forum.pjrc.com/threads/70526-__has_include(-lt-SD-h-gt-)-how-is-it-supposed-to-work

Fails the same way on 1.8.19 currently Teensyduino release

Sort of related to: #1650

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details

[matthijskooijman]

This is unfortunately a gcc bug, see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80753

The Arduino build relies on error messages for missing includes to autodetect the libraries that are used. However, the gcc bug means that if you use _has_include() on a header that is not available (like SD.h in your breaking example), then gcc will not emit an error message when the file is later included unconditionally, so arduino-cli never finds out that SD.h is needed.

The only workaround I know about is what you already showed: Make sure SD.h is included before the _has_include(). A proper fix must be done in gcc, but there's no real movement there, unfortunately.

[per1234]

Thanks @KurtE for your excellent report and thanks also matthijskooijman for the excellent evaluation. Since this issue is outside of Arduino's control, I'll close the issue.

Hopefully we will eventually see a fix in GCC and Arduino boards platforms updated to use the version containing such a fix in their toolchain. Until then, I'm sure this issue will serve as a valuable resource for others who run into the same limitation.