Error in Prism::LexCompat#insert_on_sp

[tompng]

Two different errors in v1.9.0 and in main branch

Prism.lex_compat('scan/\p{alpha}/')
# /path/to/prism/lib/prism/lex_compat.rb:811:in 'block in Prism::LexCompat#insert_on_sp': undefined method 'empty?' for nil (NoMethodError)
#             new_tokens << [[sp_line + 1, 0], :on_sp, second_whitespace, prev_token_state] unless second_whitespace.empty?
#                                                                                                                   ^^^^^^^

Prism.lex_compat('if;)')
# /path/to/prism/lib/prism/lex_compat.rb:823:in 'Prism::LexCompat#insert_on_sp': undefined method 'location' for nil (NoMethodError)
#         end_offset = eof_token.location.end_offset
#                               ^^^^^^^^^

[Earlopain]

#3900 will fix this. I'm curious, is the code/repo you happened to find this with open source? There is much syntax valid code but not much syntax invalid samples.

[tompng]

I found scan/\p{alpha}/ while editing code in IRB (copy-paste a code and partially delete with alt + delete).
My local IRB had a debug code that uses both Ripper::Lexer and Prism::Translation::Ripper::Lexer (to check IRB's ci failure in truffleruby, and I forgot to erase that debug code for a while)
Another one if;) is found by a fuzzing script below (final check before filing this issue)

tokens = ((32..127).map(&:chr)-[*'b'..'z',*'B'..'Z']+%w[do if end def class])
errors = Set.new

loop do
  code = 8.times.map{tokens.sample}.join
  begin
    Prism::Translation::Ripper::Lexer.new(code).scan
  rescue
    key = [$!.message, $!.backtrace[0]]
    unless errors.include?(key)
      p(code)
      pp $!
      errors.add(key)
    end
  end
end

[Earlopain]

Ah, right. Fuzzing makes sense. I checked the translator against the errors snippets we have in this repo and it doesn't crash on any of them (anymore). Thanks for the report (: