制作数据集（二）：使用Bing，制作更快，更干净的数据集！
背景：上一个版本的数据集制作方法尽管有效，但是数据集并不干净，例如出现很多广告等等，所以使用必应浏览器制作数据。注意，尽管必应浏览器不用FQ，但是是需要注册一些信息，不过不麻烦！
方法：首先还是先PO出作者的原文：https://www.pyimagesearch.com/2018/04/09/how-to-quickly-build-a-deep-learning-image-dataset/
- 到微软的Bing Image Search API中注册账号，它是7天免费的：https://azure.microsoft.com/en-us/services/cognitive-services/bing-image-search-api/
- 成功以后，你可以看到如下页面，并记住你的密钥1和密钥2
- 建议阅读下面两个文档，尤其是第一个文档的.json文件格式，和第二个文档的count和offset的意义，它能帮助你在后面的代码阅读中更快地理解代码的意义。
- https://docs.microsoft.com/en-us/azure/cognitive-services/bing-image-search/quickstarts/python
- https://docs.microsoft.com/en-us/azure/cognitive-services/Bing-Web-Search/paging-search-results
- 下面，在你的虚拟环境中，安装requests包
-
$ pip install requests
- 新建一个“search_bing_api.py”文件，并将下面的代码复制到你这个文件中
-
# import the necessary packages from requests import exceptions import argparse import requests import cv2 import os # construct the argument parser and parse the arguments ap = argparse.ArgumentParser() ap.add_argument("-q", "--query", required=True, help="search query to search Bing Image API for") # 这个字段表示你想搜索什么 ap.add_argument("-o", "--output", required=True, help="path to output directory of images") args = vars(ap.parse_args()) args['output'] = os.path.join(args['output'], args['query']) if not os.path.exists(args['output']): os.makedirs(args['output']) # set your Microsoft Cognitive Services API key along with (1) the # maximum number of results for a given search and (2) the group size # for results (maximum of 50 per request) API_KEY = "YOUR_API_KEY_GOES_HERE" # 将刚刚的密钥1 或 密钥2 复制到这里 MAX_RESULTS = 250 GROUP_SIZE = 50 # 最大是150，它的意义可以查看上面的文档链接2 # set the endpoint API URL URL = "https://api.cognitive.microsoft.com/bing/v7.0/images/search" # when attempting to download images from the web both the Python # programming language and the requests library have a number of # exceptions that can be thrown so let's build a list of them now # so we can filter on them EXCEPTIONS = set([IOError, FileNotFoundError, exceptions.RequestException, exceptions.HTTPError, exceptions.ConnectionError, exceptions.Timeout]) # store the search term in a convenience variable then set the # headers and search parameters term = args["query"] headers = {"Ocp-Apim-Subscription-Key" : API_KEY} params = {"q": term, "offset": 0, "count": GROUP_SIZE} # make the search print("[INFO] searching Bing API for '{}'".format(term)) search = requests.get(URL, headers=headers, params=params) search.raise_for_status() # grab the results from the search, including the total number of # estimated results returned by the Bing API results = search.json() estNumResults = min(results["totalEstimatedMatches"], MAX_RESULTS) print("[INFO] {} total results for '{}'".format(estNumResults, term)) # initialize the total number of images downloaded thus far total = 0 # loop over the estimated number of results in `GROUP_SIZE` groups for offset in range(0, estNumResults, GROUP_SIZE): # update the search parameters using the current offset, then # make the request to fetch the results print("[INFO] making request for group {}-{} of {}...".format( offset, offset + GROUP_SIZE, estNumResults)) params["offset"] = offset search = requests.get(URL, headers=headers, params=params) search.raise_for_status() results = search.json() print("[INFO] saving images for group {}-{} of {}...".format( offset, offset + GROUP_SIZE, estNumResults)) # loop over the results for v in results["value"]: # try to download the image try: # make a request to download the image print("[INFO] fetching: {}".format(v["contentUrl"])) r = requests.get(v["contentUrl"], timeout=30) # build the path to the output image ext = v["contentUrl"][v["contentUrl"].rfind("."):] p = os.path.sep.join([args["output"], "{}{}".format( str(total).zfill(8), ext)]) # write the image to disk f = open(p, "wb") f.write(r.content) f.close() # catch any errors that would not unable us to download the # image except Exception as e: # check to see if our exception is in our list of # exceptions to check for if type(e) in EXCEPTIONS: print("[INFO] skipping: {}".format(v["contentUrl"])) continue # try to load the image from disk image = cv2.imread(p) # if the image is `None` then we could not properly load the # image from disk (so it should be ignored) if image is None: print("[INFO] deleting: {}".format(p)) os.remove(p) continue # update the counter total += 1
$ mkdir dataset
$ mkdir dataset/charmander $ python search_bing_api.py --query "charmander" --output dataset/charmander # 传入希望搜索的字段，以及输出目录 [INFO] searching Bing API for 'charmander' [INFO] 250 total results for 'charmander' [INFO] making request for group 0-50 of 250... [INFO] saving images for group 0-50 of 250... [INFO] fetching: https://fc06.deviantart.net/fs70/i/2012/355/8/2/0004_c___charmander_by_gaghiel1987-d5oqbts.png [INFO] fetching: https://th03.deviantart.net/fs71/PRE/f/2010/067/5/d/Charmander_by_Woodsman819.jpg [INFO] fetching: https://fc05.deviantart.net/fs70/f/2011/120/8/6/pokemon___charmander_by_lilnutta10-d2vr4ov.jpg ... [INFO] making request for group 50-100 of 250... [INFO] saving images for group 50-100 of 250... ... [INFO] fetching: https://38.media.tumblr.com/f0fdd67a86bc3eee31a5fd16a44c07af/tumblr_nbhf2vTtSH1qc9mvbo1_500.gif [INFO] deleting: dataset/charmander/00000174.gif ...
-