Bad Bots – Headless Chrome

Bad Bots – Headless Chrome

No Comments

There’s never a shortage of bad bots and unidentifiable applications that crawl websites. Are they scraping the content? Updating it for some unnamed organization’s news site? Storing an archive of it? It’s not clear, as they typically won’t identify themselves with a legitimate robot-type user agent.

One group of firewall logs recently caught my eye for a few reasons. The first reason was that, similar to my issue with OVH Hosting in a previous blog, there were numerous clients connecting simultaneously with the same user agent. At any given time, 3 to 5 of these hosts would be crawling information, like tags and posts, off of the site. Viewing the visitors live, I saw that a high percentage of the IPs below were all using the same user agent.

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/64.0.3282.119 Safari/537.36

Here’s a copy of the Firewall log where I set up a rule to do an extended browser validation using javascript:

Does anybody know the purpose and source of these connections? Did you end up here by searching of the IPs? All of the subnets below belong to Amazon Technologies and could possibly be connected behind the scenes on Amazon Web Services.

100+ IP Addresses recorded in the month of July:

18.236.120.18
18.236.243.214
18.237.41.164
18.237.61.143
18.237.123.0
34.208.40.36
34.208.92.220
34.208.141.124
34.208.235.48
34.209.44.200
34.209.114.64
34.209.227.101
34.210.78.254
34.210.100.217
34.210.221.104
34.211.25.220
34.211.190.187
34.211.227.196
34.212.71.188
34.212.116.241
34.212.131.138
34.214.150.53
34.215.152.137
34.216.26.43
34.217.14.63
34.217.50.13
34.217.107.188
34.218.250.187
34.219.11.198
34.219.39.87
34.219.92.251
34.219.141.108
34.219.193.58
34.219.225.182
34.220.16.137
34.220.59.162
34.220.80.254
34.220.103.78
34.220.148.196
34.220.188.241
34.220.199.88
34.220.224.29
34.221.7.4
34.221.22.134
34.221.32.36
34.221.58.141
34.221.77.132
34.221.142.89
34.221.164.175
34.221.241.244
34.221.242.167
35.160.27.133
35.160.98.44
35.161.21.171
35.162.116.37
35.164.15.117
35.164.69.206
35.164.100.206
35.165.242.232
35.166.95.89
35.166.178.125
35.172.212.99
52.10.12.227
52.12.129.255
52.13.68.33
52.13.80.33
52.25.232.118
52.27.65.70
52.34.53.176
52.35.81.218
52.35.124.32
52.36.59.177
52.38.5.86
52.38.39.61
52.40.23.116
52.40.76.8
52.41.164.108
52.89.45.141
54.68.182.6
54.70.12.254
54.70.144.155
54.148.14.116
54.149.73.177
54.184.19.153
54.185.147.189
54.186.70.168
54.187.36.97
54.187.196.207
54.190.184.2
54.191.111.154
54.191.111.220
54.191.197.179
54.200.246.200
54.201.191.42
54.201.229.227
54.202.84.215
54.202.248.143
54.212.211.34
54.213.15.127
54.213.61.104
54.213.242.152
54.218.1.204
54.218.84.30
54.218.112.201
54.244.15.175
54.244.37.100
54.245.26.75
54.245.183.44
91.213.143.248
167.99.167.226