It's not immediately clear that "URL syntax" and "URL parser" conflict

[domenic]

URL syntax is a model for valid URLs---basically "authoring requirements". The URL parser section allows parsing URLs which do not follow URL syntax.

An easy example is https://////example.com, which is disallowed because the portion after https: contradicts

A scheme-relative URL
must be "//", followed by a host, optionally followed by ":" and a port, optionally followed by a path-absolute URL.

/cc @bagder via some Twitter confusion this morning.

[bagder]

Is there really any reason for accepting more than two slashes for non-file: URLs? I mean apart from this spec saying that the parser should accept them.

[domenic]

The fact that all browsers do.

[bagder]

The fact that all browsers do.

I tested Safari on a recent OS X version and it doesn't even accept three slashes. Not in the address bar and not in Location: headers in a redirect. It handles one or two slashes, no more. So I refute your claim.

The fact that all browsers do

That's exactly the sort of mindset that will prevent the WHATWG URL spec to ever become the universal URL spec. URLs need to be defined to work in more contexts than browsers.

[annevk]

URLs need to be defined to work in more contexts than browsers.

They are, no? Handling multiple slashes or not seems irrespective of that. If Safari does not do it there might be wiggle room, or Safari might hit similar compatibility issues to curl.

[bagder]

That's a large question and too big of a subject for me to address fully here.

URLs in the loose sense of the term are used all over the place.

URLs by the WHATWG definition are probably not used by much else than a handful of browsers, no. In my view (wearing my curl goggles), there are several reasons why we can't expect that to change much short-term going forward either. Like this slash issue shows.

I would love a truly universal and agreed URL syntax, but in my view we've never been further away from that than today.

[domenic]

I'm sorry for the imprecision. We often use "all browsers" to mean "the consensus browser behavior, modulo minor deviations and bugs."

The URL Standard defines URLs for software that wants to be compatible with browsers, and participate in the ecosystem of content which produces and consumes URLs meant for browsers. If cURL does not want to be part of that ecosystem, then yes, the URL Standard is probably not a good fit for cURL. But we've found over time that most software (e.g. servers which which to interact with browsers, or scraping tools which wish to be able to scrape the same sites as browsers visit) wants to converge on those rules.

[bagder]

We often use "all browsers" to mean "the consensus browser behavior, modulo minor deviations and bugs."

This made me also go and check IE11 on win7, and you know what? It doesn't support three slashes either.

To me, this is important. It shows you've added a requirement to the spec that a notable share of browsers don't support. When I ask about why (because it really makes no sense to me), you make a recursive answer and say you did this because "all browsers" act like this. Which we now know isn't true. It's just backwards in so many levels.

If cURL does not want to be part of that ecosystem

Being part of that ecosystem does not mean that I blindly just suck up what the WHATWG says a URL is without me questioning and asking for clarification and reasoning. Being here, asking questions, responding, complaining, is part of being in the ecosystem.

curl already is and has been part of the ecosystem since a very long time. Deeply, firmly and actively - we have supported and worked with URLs since back when they were still truly standard "URLs" (RFC 1738). I'm here, writing this, because I'd rather want an interoperable world where we pass URLs back and forth and we agree on what they mean.

When you actively decide to break RFC 3986 and in extension RFC 7231 for the Location: header I would prefer you could explain why. If you want to be a part of the ecosystem.

the URL Standard is probably not a good fit for cURL

I wish we worked on a URL standard, then I'd participate and voice my opinions like I do with some other standards work. A URL standard is very much a good idea for curl and for the entire world.

A URL works in browsers and outside of browsers. It can be printed on posters, it works parsed and highlighted by terminal emulators or IRC clients, they get parsed by scripts and they get read out loud over the phone by kids to their grandparents. URLs are, or at least could be, truly universal. Limiting the scope to "all browsers" limits the usability of them. It fragments what a URL is and how it works (or not) in different places and for which uses.

If you want a URL standard, you must look beyond "all browsers".

[domenic]

This made me also go and check IE11 on win7, and you know what? It doesn't support three slashes either.

Edge does, however: http://software.hixie.ch/utilities/js/live-dom-viewer/?saved=4182

In general, Edge has made changes like this to be compatible with the wider ecosystem of web content. I can't speak for their engineers, but this shows clear convergence.

---

It's good to hear you're interested in participating. That wasn't my impression from your earlier comments, and I welcome the correction.

[JohnMH]

Why should malformatted URLs be parsed? Surely the solution is to simply tell people who are using malformatted URLs to.. stop using malformatted URLs?

[jyasskin]

In the interest of looking for ways forward, instead of just saying "no", per https://twitter.com/yoavweiss/status/730173495464894465, it might make sense to collect usage data and see if browsers can simplify the URL grammar.

[JohnMH]

It may be best to ignore that browsers even use URLs, because there are definitely other pieces of software that use URLs. Consider the following URL: irc://network:port/#channel

On May 10, 2016 7:31:35 PM EDT, Jeffrey Yasskin notifications@github.com wrote:

In the interest of looking for ways forward, instead of just saying
"no", per https://twitter.com/yoavweiss/status/730173495464894465, it
might make sense to collect usage data and see if browsers can simplify
the URL grammar.

---

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#118 (comment)

John M. Harris, Jr.
PGP Key: f2ea233509f192f98464c2e94f8f03c64bb38ffd

Sent from my Android device. Please excuse my brevity.

[domenic]

I'd suggest the following plan to any browsers interested in tightening the URL syntax they accept:

• look through https://github.com/w3c/web-platform-tests/blob/master/url/urltestdata.json and find all non-error results that you wish became error results
• (optionally, wait until I finish my long-delayed project to expand that file to give 100% coverage of the spec. Currently it gives around 60%.)
• Instrument all URL parsing to figure out how often these undesirable patterns occur
• When the numbers come back, decide which percentage of users or pages you are willing to break, and pick the subset of algorithm changes you can make to do so up to that percentage
• optionally, weigh the percent of users broken vs. the corresponding spec or implementation complexity reduction
• start coordinating with other vendors to see which of your changes they're interested in. (Some cases might already behave differently in those other vendors' browsers, which could help!)
• Ship, preferably in a coordinated fashion with appropriate devrel support.

[JohnMH]

Browsers are not the only applications that use URLs.

On May 10, 2016 7:40:56 PM EDT, Domenic Denicola notifications@github.com wrote:

I'd suggest the following plan to any browsers interested in tightening
the URL syntax they accept:

• look through
  https://github.com/w3c/web-platform-tests/blob/master/url/urltestdata.json
  and find all non-error results that you wish became error results
• (optionally, wait until I finish my long-delayed project to expand
  that file to give 100% coverage of the spec. Currently it gives around
  60%.)
• Instrument all URL parsing to figure out how often these undesirable
  patterns occur
• When the numbers come back, decide which percentage of users or pages
  you are willing to break, and pick the subset of algorithm changes you
  can make to do so up to that percentage
• optionally, weigh the percent of users broken vs. the corresponding
  spec or implementation complexity reduction
• start coordinating with other vendors to see which of your changes
  they're interested in. (Some cases might already behave differently in
  those other vendors' browsers, which could help!)
• Ship, preferably in a coordinated fashion with appropriate devrel
  support.

---

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#118 (comment)

John M. Harris, Jr.
PGP Key: f2ea233509f192f98464c2e94f8f03c64bb38ffd

Sent from my Android device. Please excuse my brevity.