User Agent Header – Web Development
Articles,  Blog

User Agent Header – Web Development


>>Okay, so I want to add a little more about user agents. So it’s, it’s one of the most important headers in an HTTP request. And when we were doing Reddit, user agents were really important to us. So we had the site that was online and really popular. And users were always you know, often writing scripts to, to pull content out on Reddit. And, you know, mostly they were doing good things. They were building you know, tools to, you know, do data collections so they can do some, you know, cool blog post about how Reddit works and that sort of thing. Sometimes they’re doing bad things. You know, there are spammers, you know, looking for weaknesses or, you know, looking for ways to game the system. And user agents are important because sometimes, you know, a user would hit us a little too hard too fast, you know, basically, you know, hurting the website for real users. And if they had a legitimate user agent, we could look at them. We can you know, rate limit them accordingly or we can reach out to them and say, hey you know, you are hurting us and you know, they, they would adjust, and it’s cool. The Googlebot is a really good example of this. The Googlebot is Google’s WebCrawler that indexes website for, you know, their search engine. Googlebot would just punish us. They would hit us as fast as they could, and sometimes that would actually slow down the site. And until we learned how to handle that properly, we would just, you know, rate limit them a little bit. And, you know, Googlebot, you know, their feelings weren’t hurt, they can still index the site. We just made them do it slower. But when people show up with fake user agents and we can tell that, you know, this one IP and this one user agent is just pummeling us and, and they’re trying to pretend to be a browser, we would just block them altogether. So, using good user agents when you’re writing software that, interacts with other people’s websites is, is a really nice, courteous thing to do. And it’s one of the things that makes the web work well for everybody. So it’s always important to have a nice accurate user agent and to be honest when you can.>>Okay, now that I’d like you to do is promise to always use user agents appropriately when you’re writing your web software.

2 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *