This example shows how to use the Beautiful Soup library to find all images referenced in a bunch of html files, then filter to a particular size range – this works well to take out header images, logos, tracking pictures, etc. This assumes a system where you mirrored a website’s directory structure with wget. Unlike most examples this handles some real data – e.g. 404’ed images and different URL structures.<\/p>\n
\nfrom bs4 import BeautifulSoup\nimport Image\nimport urllib2\nimport os, re\n \ndomain = \"www.example.com\"\nindir = \"\"\nfre = re.compile(\".*html$\")\n\nfor root, dirs, filenames in os.walk(indir):\n for f in filenames:\n if fre.match(f):\n htmlfile = os.path.join(root, f)\n print \"Parsing \" + htmlfile\n \n soup = BeautifulSoup(open(htmlfile), \"html5lib\")\n \n images = soup.find_all('img')\n def f(x) : \n if (x['src'] is None):\n return False\n\n imgSrc = x['src']\n\n if (\"http:\/\/\" in imgSrc):\n imgSrc = imgSrc[7:]\n\n if domain not in imgSrc:\n imgSrc = os.path.join(root, imgSrc)\n\n imgFile = imgSrc\n\n try: \n img = Image.open(imgFile)\n width, height = img.size\n return width > 150 and height > 150\n except IOError:\n return False\n\n art = filter(f, images)\n print(art)\n<\/pre>\n","protected":false},"excerpt":{"rendered":"This example shows how to use the Beautiful Soup library to find all images referenced in a bunch of html files, then filter to a particular size range – this works well to take out header images, logos, tracking pictures, etc. This assumes a system where you mirrored a website’s directory structure with wget. Unlike … <\/p>\n