[Flang][Preprocessor] Avoid creating an empty token when a kind suffix is torn by a pasting operator

[rofirrim]

This input "tears" the expected tokens of an integer-literal due to a pasting operator ##. When lexing 1_## we generate the sequence of tokens ['1_', ''], the second being an empty token of length zero. The second token is created at the end of Prescanner::NextToken.

Creating an empty token by accident (due to two consecutive CloseToken without consuming anything) can cause TokenSequence::pop_back to assert.

If zero-length tokens are acceptable, then instead of this patch we may have to fix the logic in TokenPasting found in preprocessor.cpp.

[llvmbot]

@llvm/pr-subscribers-flang-parser

Author: Roger Ferrer Ibáñez (rofirrim)

Changes

This input "tears" the expected tokens of an integer-literal due to a pasting operator ##. When lexing 1_## we generate the sequence of tokens ['1_', ''], the second being an empty token of length zero. The second token is created at the end of Prescanner::NextToken.

Creating an empty token by accident (due to two consecutive CloseToken without consuming anything) can cause TokenSequence::pop_back to assert.

If zero-length tokens are acceptable, then instead of this patch we may have to fix the logic in TokenPasting found in preprocessor.cpp.

---

Full diff: https://github.com/llvm/llvm-project/pull/139795.diff

2 Files Affected:

• (modified) flang/lib/Parser/prescan.cpp (+8)
• (added) flang/test/Preprocessing/torn-token-pasting-1.F90 (+9)

diff --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 3bc2ea0b37508..004e4f013f90a 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -937,6 +937,7 @@ bool Prescanner::HandleKindSuffix(TokenSequence &tokens) {
   if (*at_ != '_') {
     return false;
   }
+  auto underscore = *at_;
   TokenSequence withUnderscore, separate;
   EmitChar(withUnderscore, '_');
   EmitCharAndAdvance(separate, '_');
@@ -951,6 +952,13 @@ bool Prescanner::HandleKindSuffix(TokenSequence &tokens) {
   }
   withUnderscore.CloseToken();
   separate.CloseToken();
+  // If we only saw "_" and nothing else, we have handled enough but we do not
+  // want to close the token here, or we will generate an extra token of length
+  // zero.
+  if (separate.SizeInTokens() == 1) {
+    EmitChar(tokens, underscore);
+    return true;
+  }
   tokens.CloseToken();
   if (separate.SizeInTokens() == 2 &&
       preprocessor_.IsNameDefined(separate.TokenAt(1)) &&
diff --git a/flang/test/Preprocessing/torn-token-pasting-1.F90 b/flang/test/Preprocessing/torn-token-pasting-1.F90
new file mode 100644
index 0000000000000..5e080129a94d1
--- /dev/null
+++ b/flang/test/Preprocessing/torn-token-pasting-1.F90
@@ -0,0 +1,9 @@
+! RUN: %flang -E %s 2>&1 | FileCheck %s
+! CHECK: IF(10>HUGE(1_4).OR.10<-HUGE(1_4)) CALL foo()
+#define CHECKSAFEINT(x,k)  IF(x>HUGE(1_  ##  k).OR.x<-HUGE(1_##k)) CALL foo()
+
+program main
+  implicit none
+
+  CHECKSAFEINT(10, 4)
+end program main