drafts.csswg.org keeps blocking my IP, rate limits are too low #11354
Comments
|
I can access it if I use the data connection from my phone. But I can't access it from the my home's connection. Maybe my IP is banned or something, I'm behind CGNAT :( |
|
It is working for me. I have been banned in the past though for requesting drafts too quickly :( |
|
Works again for me. Comparing |
|
You most likely triggered a CrowdSec block. This can happen for exceeding rate limits (among other things) and will block all traffic from your IP to the server for 4 hours. If it keeps happening let me know what your IP address is and I can look at what caused the block, and adjust rate limits if needed. It's a balance between allowing normal access and blocking scrapers. |
|
Individual drafts still work, but the overall drafts index is down. |
|
Still down. Individual drafts do work if you know the address, but the overall index is still missing. |
|
@plinss My IP seems blocked again, it's 77.75.177.5, I wonder if I'm sharing it with a bot due to CGNAT. |
|
Your IP was blocked due to rate limiting violations. The only UA string associated with that IP in the logs is "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0", so doesn't look like a bot, Just hitting too many specs too close together (each image load counts, but it does allows higher bursts for those). If you get a 429 error, just back off for a while to avoid an IP block. |
|
I didn't get see any error before getting blocked, though. |
|
Blocked again, same IP, these limits seem way too low. |
|
And blocked again. Pretty sure I only loaded https://drafts.csswg.org/css2/ and https://drafts.fxtf.org/filter-effects-2/. Then blocked when trying to load https://drafts.csswg.org/css-flexbox-1/ |
Loirooriol
changed the title
|
OK, ironically after whatwg/html#11005, I need to switch back to using Github pages, e.g. https://w3c.github.io/csswg-drafts/css-flexbox/ I have written this violentmonkey script to avoid the redirection to the official website: // ==UserScript==
// @name CSSWG drafts in Github Pages
// @namespace https://github.com/Loirooriol/
// @match https://w3c.github.io/csswg-drafts/*
// @grant none
// @version 1.0
// @author Oriol Brufau
// @description Avoids redirecting away from the CSSWG drafts hosted in Github Pages
// @run-at document-start
// ==/UserScript==
addEventListener("beforescriptexecute", function listener(event) {
if (event.target.textContent.includes("githubPrefix")) {
event.preventDefault();
removeEventListener("beforescriptexecute", listener);
}
}); |
|
And how long are these blocks?? I'm still blocked, almost 12h later. |
|
I increased the burst limit on images, this should help you while still catching crawlers. I don't see your IP on the current block list, either it expired or you're using a different IP now. Blocks are for 4 hours per violation, so the 4th time you get blocked is for 16 hours. |
|
So maybe it's something else. My IP is still 77.75.177.5, and I can't reach |
|
OK it has been fixed just now, I guess you did something, thanks! |
|
Happening again (still same IP). |
|
I've frequently been hitting this lately, but only just came across this issue. I just assumed the server was overloaded or something. It's there some way the limits can be raised generally? All I'm doing is opening multiple specs at a time, to cross-reference things. I'm also using Firefox on Linux, maybe that combination causes spammy requests? EDIT: My IP is 92.29.234.208 in case that's helpful. |
|
I brought this up with Ladybird developers and a couple of other people have been caught by this, just for trying to look at specs to implement them. Obviously I'm completely unaware of what level of scraping/bot traffic you were getting without this, so I don't know how bad it is - but if the blocking can be relaxed then we'd all appreciate it! |
|
Multiple implementor contributors to the Ladybird project have been repeatedly running into this problem. Please fix whatever the root cause of it is so that this doesn't keep happening. We don't want to keep dealing with it case by case. The CSS WG drafts are the only specs we're running into this problem with. We're able to access all other W3C specs without running into rate limiting. So all that's needed here is to minimally make the behavior for accessing CSS WG drafts be the same as for all other W3C drafts. |
|
I'd also like to request that this be added to the agenda for the next working-group meeting, and I'd be happy to join the meeting to discuss the problem and how the group plans to fix it. |
|
The "root cause" is that the server is constantly getting hit by hundreds of crawlers that ignore robots.txt. The traffic also spikes when multi-billion dollar companies, who don't contribute to the draft servers at all, post links to drafts in press and social media releases touting their latest features, or even crawl the server themselves to feed their LLMs. While the drafts are currently served by two fairly robust servers, they're not able to handle the load without multiple layers of bot defenses. Rate limiting, followed by IP blocks, has been an effective tool to that end and is not going to get turned off any time soon. The rate limits for regular drafts are currently 1 request per second with bursts of up to 50 requests, images allow bursts of up to 500. Multiple rate limit violations are required before IP blocks take effect. If Ladybird contributors are hitting that limit, then they're either behind a single NAT or proxy, or they need to rethink their draft access. If you want to provide specific IPs, I can look at the logs and see what's going on. I can also add a few IPs, or a small range to the blocking safelist (provided they're not abusing the server themselves). As far as "addressing the root cause", you can either get everyone to stop crawling the site, get someone to start sponsoring the hosting so I can scale up the servers, or find someone to take over hosting it entirely. I'm more than happy for someone else to take over, but anyone offering to do so, or replace the server with a different solution, would be best served by starting out asking what the server is really hosting and what kinds of loads it's under, before proposing half-assed solutions. I'm currently paying about $150/month out of my own pocket for the servers, for no personal gain, and have been doing so for years. And that's not taking my time into account, which has been considerable. So maybe being a bit less demanding here would also be a good place to start. |
|
On behalf of the W3C team, I am conveying that we do not consider this resolved. Please don't close this issue again until we have agreement with the working group and with the W3C team that it's actually resolved. |
|
On behalf of the un-sponsored person paying for the server, doing all the work, and currently having their time wasted, I am conveying that I consider your tone and behavior here inappropriate and unprofessional. This issue was resolved on April 2 at 1:08PM PDT when the rate limits were last adjusted. You have provided no information that anyone has experienced a rate-limit based IP block since then. What information you have provided is vague, anecdotal, and not actionable. Please do not reopen this issue without first confirming that: legitimate users (i.e. not bots) are currently blocked, those blocks have started since the rate limits were last adjusted (see above where repeated blocks result in longer bans), and providing the date and time the block started and the IP address(es) affected. Otherwise there's no actionable information and nothing to do here. If there are left over long-term blocks from before the rate limits last changed, I also need IP addresses to release those. If you want to start a general conversation with the WG about replacing the draft server, feel free to do so. I welcome it. File another issue for that, and please begin the conversation with a practical plan, a source of funding, or replacement servers, not just complaints about the status quo. This issue is about IPs being blocked due to rate limit violations. Stop hijacking it. |
@sideshowbarker See |
Good point. Do you have that information to hand? |
For a concrete example of rate limits being too-aggressively applied, I visited https://drafts.csswg.org/css-fonts-4/ at around 9:10 am UTC today from a link on mdn (https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face#formal_syntax). I immediately realized "oh, I wanted the specific link to font face lower down in the mdn page", so I closed the tab, and clicked on the link to https://drafts.csswg.org/css-fonts/#font-face-rule from https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face#specifications instead. I was hit with an immediate 429. |
|
Blocked from 98.217.98.87 at 14:27 EST 21 April 2025 while attempting to access https://drafts.csswg.org/css-color-5/#absolute-color |
|
The only log entries I find for that IP on the draft server are:
No 429's, no IP blocks. Are you sure about that IP? Also, if the rate limit defenses kicked in you'd have been blocked for a minimum of 4 hours. A 2 minute block must have been something else. |
@ADKaster I need an IP address to be able to check the logs. Also, did you just get a 429 response or was your IP blocked? |
|
I just got blocked while trying to add myself into https://wiki.csswg.org/planning/paris-2025 |
|
I released that block and increased the burst limit on the wiki (which was lower than the draft server) |
|
Yeah, I could reload the page. When trying to save, I got blocked again. |
|
It helps if I actually reload the server so the new configuration kicks in... |
|
I'm blocked again, adding myself to the wiki is impossible |
|
I increased the rate limit on the wiki some more and unblocked you |
The server doesn't respond. Not even
ping drafts.csswg.orgworks.@plinss
The text was updated successfully, but these errors were encountered: