Soft deprecation of the project?

[iliabylich]

So, Prism is out and it's:

1. correct
2. comes out of the box with MRI (and IIRC with other Ruby implementations)
3. faster (I think on TruffleRuby/JRuby it's absurdly faster because it doesn't run in interpreted mode like it does with parser)
4. always up-to-date
5. written in a language with perfect interop capabilities

On the other side backporting changes to parser becomes more and more difficult. MRI doesn't use yacc/bison anymore, instead there's Lrama that is also an LALR(1) parser generator but it has more "syntax sugar" that Racc doesn't have, and so the grammar will diverge more and more over time. And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I personally believe that parser should be soft-deprecated. What do you guys think? @kddnewton @bbatsov @koic @palkan @marcandre @mbj

And if you agree do you see anything we could do to parser to make transition easier?

[mbj]

The real use case of this gem left for me is multi version capability. Prism is not shipped with all the ruby versions my use cases still have to support, basically all non EOL (MRI) rubies.

If this gem where to stop supporting future ruby versions I'd be fine. Cannot speak for the rest of the ecosystem OFC.

[kddnewton]

In general I think this is the right long-term direction, but I'm uncomfortable soft deprecating until we mirror the tree rewriting functionality, since so many consumers use that. I think it could be added to the Ruby API of Prism relatively easily, but I haven't had time to look into this yet.

I'm not sure if there are any other APIs that I'm not thinking about, but that's the main one I know we are missing.

[bbatsov]

Indeed. I don't see how we can deprecated parser without the rewriting functionality. RuboCop also relies on the multi-version capability quite a lot, although I guess we can live without it if push comes to shove.

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases? I think that a realistic time-frame for full migration to Prism is probably 2-3 years (for the projects that are still active, that is), so parser will definitely require some maintenance is the mean time.

[iliabylich]

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases?

My initial thought was to stop doing it. At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

[mbj]

I think we need to put evolutionary pressure on and NOT release compatibility with new versions.

[bbatsov]

I'm not super fond of measures that will be quite disruptive to the end users - to me it seems just wild to stop supporting something without providing a complete alternative for it. E.g. what happens if the rewriting is still not supported by Prism when Ruby 3.5 comes out?

At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

Breaking existing apps is not exactly in the service of the community either. ;-)

I'm not arguing that parser should go away - I'm just concerned about abandoning the project prematurely and causing pain for the users of tool depending on it.

[mbj]

To play a bit of devils advocate here: How is not supporting a new future ruby release "breaking existing apps"?

[bbatsov]

Okay, "causing problems for the existing apps". I'm fairly certain RuboCop's users won't be exactly happy if we tell them we don't really support newer Ruby releases for whatever reasons. :-) And I'm not sure that simply stopping work on parser will motivate everyone to implement whatever is missing, etc. Also - how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

Moving fully to Prism's native AST is going to be quite the time investment for many projects. I'd like for us to get there, but ideally I don't want us to be pressured into a timeline for this given that everyone working on RuboCop does this in their spare time.

[mbj]

Is there a translation layer from prosm to this parsers AST somewhere?

[palkan]

Is there a translation layer from prism to this parsers AST somewhere?

Prism supports AST translation for popular parsers: https://github.com/ruby/prism/tree/main/lib/prism/translation

And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I think, deprecating Parser as soon as parse.y is deprecated makes sense. Otherwise the effort required to port Ruby changes to Parser would be enormous.

how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

From my experience, adding new nodes could be done relatively easy (compared to upgrading the grammar and lexer). I see, how, in theory, we can use Prism for newer Ruby versions as a parser and a Parser-compatible AST builder.

So, if we can keep Parser Ruby internals and APIs work in new Ruby versions (without providing the corresponding syntax support), that would smooth the transition.

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

[iliabylich]

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

Yes, that's more or less what I proposed (probably the word "deprecate" was a bit misleading). I'm not proposing to yank parser from rubygems.org, only to stop backporting new grammar changes.

I personally never touched rewriting API, shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Breaking existing apps is not exactly in the service of the community either. ;-)

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

[palkan]

shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Yeah, it works right now. Well, at least, for Ruby Next.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

[mbj]

Is there an unparser equivalent yet?

[kddnewton]

Truthfully I don't know if parse.y is going to be deprecated, or if that will happen any time soon at all. I imagine it could stick around for quite a long time. That being said, it is getting further away from strict bison/racc syntax, so I know it's quite a maintenance burden to hold on to.

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

I think this would work, but we will want to test the translation layer even more. I think there are issues with string escapes at the moment, but I'm not sure.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

That would be really great.

Is there an unparser equivalent yet?

I haven't managed to get it over the finish line yet, but yes https://github.com/ruby-syntax-tree/syntax_tree-prism is that work. It's almost done.

[iliabylich]

Is there an unparser equivalent yet?

Probably no, but I've found "untokenizer" made by @koic 😄

[bbatsov]

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

To me it seems that'd be quite painful, especially if Prism doesn't support specifying target Ruby versions. And by "break" I mostly refer to "break the way" things are happening right now, not so much actually breaking existing apps. (and force clients to take some steps to ensure their tools will work with future Rubies that are not supported by parser) Anyways, I guess me and the rest of our team will have some thinking to do about the future and the next steps on our end.

To summarize, for us the biggest problems right now are:

• RuboCop allows users to specify which Ruby version they are targeting, regardless of the Ruby runtime and that's not currently supported by Prism (this was actually one of the things about parser I liked the most when I migrated RuboCop from ripper to parser in the early days of the project)
• We rely heavily on the re-writing and ideally this should be added to Prism. Otherwise we might consider pulling it into rubocop-ast or something along those lines

I'm guessing that @koic and @marcandre will have more to add on the subject.

[mbj]

Probably no, but I've found "untokenizer" made by @koic 😄

unfortunately not enough for my use case.

@bbatsov Okay so I saw a world where parser would stop to add new ruby version support, and everyone who needs newer ruby version support would than conditionally use Prism via a compatibiltiy layer. But your use case of running one ruby version and targeting another breaks this option.

All my use cases would be covered but not yours, curious if there are plans for a version target @kddnewton?

[marcandre]

I think the rewriter could definitely be included in rubocop-ast, or even better be a separate gem altogether.

[koic]

Much of what @bbatsov has already said aligns with my own thoughts, and I completely agree with his perspective. Specifically, I also share concerns about the impact on RuboCop users who rely on the Parser gem.

Direct Support for Prism Is Not Straightforward

First, I think the path to directly supporting Prism without going through Prism::Translation::Parser is quite challenging for RuboCop.

For instance, node pattern would require new support to describe the Prism AST, which differs from that of the Parser gem AST. Moreover, ensuring a seamless transition for users likely means supporting existing implementations written using the traditional Parser gem AST in parallel. So, It's important to note that changes in AST programming will affect not only RuboCop core developers but also its gem users.

And, node pattern is just one example, and there are some other aspects of design and implementation to consider.

The Importance of Prism::Translation::Parser

In reality, the transition will likely involve maintaining a Parser gem-compatible API via Prism::Translation::Parser for a while, gradually moving forward.

If the goal is to slowly wind down development of the Parser gem, then enhancing maintainability through Prism::Translation::Parser rather than starting with vanilla Prism would be a more practical approach, especially considering tools heavily dependent on the Parser gem.

It might be possible to design a transition where older Ruby versions continue to use the Parser gem, while newer Ruby versions rely on Prism::Translation::Parser, minimizing the impact on users.

So, if Prism::Translation::Parser becomes stable, even if the Parser gem stops tracking Ruby's parse.y in the future, Ruby syntax changes could still be handled through Prism::Translation::Parser.

Summary

Either way, since the future remains uncertain, I think compatibility maintaining Parser gem and Prism::Translation::Parser should be a priority for now.

[kddnewton]

All my use cases would be covered but not yours, curious if there are plans for a version target @kddnewton?

We have version targeting built into Prism right now (you can pass version: into the parse* family of functions).

[iliabylich]

I'm personally not going to backport changes anymore. If you want to keep this project up-to-date feel free to do so. I can still bump versions for existing ..3.3.X versions and fix potential deprecation warnings (for a few years or so, I understand that there are people that are stuck with older Rubies).

[bbatsov]

@iliabylich Fair enough. I definitely wouldn't expect you to be doing work you don't consider meaningful. I'm not sure who from our team has access to the repo, but it might be a good idea to make some of us co-maintainers, so we can keep supporting it as long as necessary for the needs of RuboCop.

[dgollahon]

All my use cases would be covered but not yours, curious if there are plans for a version target @kddnewton?

We have version targeting built into Prism right now (you can pass version: into the parse* family of functions).

@kddnewton It seems like this only targets 3.3+.. But 3.2 is still under normal maintenance and 3.1 is under security maintenance.

It seems kind of tough to transfer from parser to prism when you either have to support < 3.4 on parser or >= 3.3 on prism. I know this is part of why the translation layer exists but it's kind of a weird state of affairs to have to support both tools v.s. switch to the new prism APIs.

--

There's also been some discussion about the rewriter above--I think also having that functionality either native to prism or maybe split out would make adoption easier as well.

[whitequark]

but it might be a good idea to make some of us co-maintainers, so we can keep supporting it as long as necessary for the needs of RuboCop.

I will add anybody with a reasonable track record to the maintainer and gem uploader list.

[bbatsov]

@whitequark How about me and @koic?

[bbatsov]

(just a note - he has a different handle there https://rubygems.org/profiles/koic_ito)

[koic]

@bbatsov Thank you. I was previously granted release privileges along with commit privileges in #608 (comment).

[bbatsov]

@koic So I guess you can help with the list of PRs that @Earlopain submitted and I'd like for us to move along somehow. I think he's also a good candidate for a co-maintainer for parser.

[whitequark]

@bbatsov You have the commit bit and the gem publish rights.

[koic]

Yeah, I can see the currently open PRs, but I'm starting to have some concerns about whether maintaining both Parser and Prism::Translation::Parser is truly the best approach.

As you know, starting with Ruby 3.4, Prism has become the standard parser for Ruby, and it will likely continue to support any new syntax introduced in Ruby 3.5 and beyond. When that happens, Prism::Translation::Parser will need to be updated accordingly.

On the other hand, as @iliabylich mentioned, the parse.y files used by (alternative) CRuby’s parser generator Lrama and the one used in the Parser gem with Racc are diverging.

At this point, while it is not impossible to keep tracking CRuby’s parse.y, I am still uncertain whether the cost is justified.

As a result, the implementation of the it parameter syntax in Ruby 3.4 has also been opened a PR precede in Prism::Translation::Parser, as it has been easier to support there:
ruby/prism#3481

As shown in the PR, the AST designed by the Parser gem is highly compatible with RuboCop's needs, and Parser::Translation::Parser is positioned as an extension that advances ahead of it.

Prism (or Lrama) was intended to simplify parser maintenance with a Universal Parser, yet the current reality involves maintaining multiple parsers. How to interpret this situation is under consideration. Of course, no decision has been made at this point.

[bbatsov]

@koic Long-term - no arguments from me. But I'm guessing it'd be prudent to maintain parser to some degree in the short term (e.g. for Ruby 3.4 and 3.5) until more people have switched to Prism.

[bbatsov]

Also it seems to me that @Earlopain's improvements are non-controversial and if the work is done we might as well ship it. :-)

[Earlopain]

I think he's also a good candidate for a co-maintainer for parser

Thanks for your word of confidence, I woudn't mind it.

I made the PRs after I saw this issue here to see how much effort it would be to make it fully compatible with 3.4. I quickly realized that no, I won't be able to make that work. I wouldn't mind seeing the bugfixes merged since I already put in the work.

The ones specific to 3.4 syntax, merging them would give a wrong impression about intentions to support it in the future, which I don't want to give anymore. In my opinion, these should be closed.

[bbatsov]

The ones specific to 3.4 syntax, merging them would give a wrong impression about intentions to support it in the future, which I don't want to give anymore. In my opinion, these should be closed.

Got it. I was hoping that those were indication that it supporting 3.4 would be doable. Anyways, I guess we'll focus on getting the bugfixes out, as they would benefit everyone.

I'll update the README here accordingly when I get the chance (e.g. soft deprecation, etc) and we'll need to be clearer in https://docs.rubocop.org/rubocop/compatibility.html about the need for people to use Prism if they need to target Ruby 3.4+.

[Earlopain]

Got it. I was hoping that those were indication that it supporting 3.4 would be doable

Surely it is for someone with more expertise in ragel and parsers in general but I am not that person. All my changes were possible in ruby without needing to touch the syntax directly (which looks daunting to say the least).

I'll update the README here accordingly when I get the chance (e.g. soft deprecation, etc) and we'll need to be clearer in https://docs.rubocop.org/rubocop/compatibility.html about the need for people to use Prism if they need to target Ruby 3.4+.

I don't think there's a need for that. Currently, it works very fine with parser because the syntax barely changed and once ruby/prism#3481 is merged and released rubocop-ast can just hard-depend on prism and simply make that choice for the user without anyone needing to explicitly opt in. It can only parse Ruby starting from 3.3 but has no trouble running on 2.7 which works out very well for RuboCop in that regard

[koic]

I have also been thinking about https://docs.rubocop.org/rubocop/compatibility.html page. At present, for most users, there is no significant difference between the Parser gem and Prism::Translation::Parser. From a compatibility standpoint, the Parser gem has a slight advantage.

However, with the next release of the Prism gem, reported incompatibilities in Prism::Translation::Parser are expected to be improved. Furthermore, once it block parameter syntax support from ruby/prism#3481 is included, Prism::Translation::Parser will provide more modern support for Ruby 3.4 users.

That will likely be one of the key points for making a thinking for RuboCop's backend parser.