今天盘点 6 个爬虫开源项目，它们可以帮你爬天爬地怕空气，爬微博、爬B站、爬知乎、爬*站。[scode type="share"]提前声明，切勿使用这些项目从事非法商业活动，仅用于用于科研学习图片[/scode]01微博爬虫这个开源项目程序可以持续爬取一个或多个新浪微博用户（如李文di、无疫烦）的数据，并将结果信息写入文件或数据库。写入信息几乎包括用户微博的所有数据，包括用户信息和微博信息两大类。地址：https://github.com/dataabc/weiboSpider 爬取结果可写入文件和数据库，具体的写入文件类型如下：txt文件csv文件json文件MySQL数据库MongoDB数据库SQLite数据库同时支持下载微博中的图片和视频，具体的可下载文件如下：原创微博中的原始图片转发微博中的原始图片原创微博中的视频转发微博中的视频原创微博Live Photo中的视频转发微博Live Photo中的视频首先需要修改 config.json 文件，然后爬取，程序会自动生成一个 weibo 文件夹，我们以后爬取的所有微博都被存储在这里。然后程序在该文件夹下生成一个名为"微博名字"的文件夹，明星的所有微博爬取结果都在这里。文件夹里包含一个csv文件、一个txt文件、一个json文件、一个img文件夹和一个video文件夹，img文件夹用来存储下载到的图片，video文件夹用来存储下载到的视频。如果你设置了保存数据库功能，这些信息也会保存在数据库里，数据库设置见设置数据库部分。02Python爬虫教程Python爬虫教程系列、从 0 到 1 学习 Python 爬虫，包括浏览器抓包，手机 APP 抓包，如 fiddler、mitmproxy，各种爬虫涉及的模块的使用，如：requests、beautifulSoup、selenium、appium、scrapy 等，以及验证码识别，MySQL，MongoDB 数据库的 Python 使用，多线程多进程爬虫的使用，css 爬虫加密逆向破解，JS爬虫逆向，分布式爬虫，爬虫项目实战实例等。地址：https://github.com/wistbean/learn_python3_spider 03爬虫集合这个开源项目收集了各种爬虫 ，包括 Blibli、博客园、百度百科、北邮人、百度云网盘、Boss、贝壳、豆瓣、CSDN、抖音、GitHub、京东、知乎、拉钩、链家、微信公众号、网易云等等，你能想到的国内外网站爬虫，都可以先来这里看看有没有开源的爬虫。地址：https://github.com/facert/awesome-spider 04智能爬虫平台这个开源平台以流程图的方式定义爬虫，是一个高度灵活可配置的爬虫平台。你可以在该平台配置各种爬虫。地址：https://gitee.com/ssssssss-team/spider-flow 接下来以流程图的方式，开始配置一些变量和参数，点开始就能爬出你想要的数据。05Java爬虫Spiderman 是一个Java开源Web数据抽取工具，它能够收集指定的Web页面并从这些页面中提取有用的数据。Spiderman主要是运用了像XPath，正则表达式等这些技术来实数据抽取。地址：https://gitee.com/l-weiwei/spiderman 06爬虫大全这个开源项目包含多种网站、电商数据爬虫。包含：淘宝商品、微信公众号、大众点评、招聘网站、闲鱼、阿里任务、scrapy博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评️️️。地址：https://gitee.com/AJay13/ECommerceCrawlers
[scode type="share"]来源：blog.csdn.net/qq_35402412/article/details/113627625[/scode]目的爬取搜狗图片上千张美女图片并下载到本地准备工作爬取地址：https://pic.sogou.com/pics?query=%E7%BE%8E%E5%A5%B3分析打开上面的地址，按F12开发者工具 - NetWork - XHR - 页面往下滑动XHR栏出现请求信息如下：Request URL ：https://pic.sogou.com/napi/pc/searchList?mode=1&start=48&xml_len=48&query=%E7%BE%8E%E5%A5%B3 分析这段请求URL的主要几个参数：start=48 表示从第48张图片开始检索xml_len=48 从地48张往后获取48张图片query=？搜索关键词（例：美女，这里浏览器自动做了转码，不影响我们使用）点击Respose，找个JSON格式器辅助过去看看。JSON格式：https://www.bejson.com/分析Respose返回的信息，可以发现我们想要的图片地址放在 picUrl里，思路通过以上分析，不难实现下载方法，思路如下：设置URL请求参数访问URL请求，获取图片地址图片地址存入List遍历List，使用线程池下载到本地代码SougouImgProcessor.java 爬取图片类import com.alibaba.fastjson.JSONObject; import us.codecraft.webmagic.utils.HttpClientUtils; import victor.chang.crawler.pipeline.SougouImgPipeline; import java.util.ArrayList; import java.util.List; /** * A simple PageProcessor. * @author code4crafter@gmail.com * @since 0.1.0 */ public class SougouImgProcessor { private String url; private SougouImgPipeline pipeline; private List<JSONObject> dataList; private List<String> urlList; private String word; public SougouImgProcessor(String url,String word) { this.url = url; this.word = word; this.pipeline = new SougouImgPipeline(); this.dataList = new ArrayList<>(); this.urlList = new ArrayList<>(); } public void process(int idx, int size) { String res = HttpClientUtils.get(String.format(this.url, idx, size, this.word)); JSONObject object = JSONObject.parseObject(res); List<JSONObject> items = (List<JSONObject>)((JSONObject)object.get("data")).get("items"); for(JSONObject item : items){ this.urlList.add(item.getString("picUrl")); } this.dataList.addAll(items); } // 下载 public void pipelineData(){ // 多线程 pipeline.processSync(this.urlList, this.word); } public static void main(String[] args) { String url = "https://pic.sogou.com/napi/pc/searchList?mode=1&start=%s&xml_len=%s&query=%s"; SougouImgProcessor processor = new SougouImgProcessor(url,"美女"); int start = 0, size = 50, limit = 1000; // 定义爬取开始索引、每次爬取数量、总共爬取数量 for(int i=start;i<start+limit;i+=size) processor.process(i, size); processor.pipelineData(); } } SougouImgPipeline.java 图片下载类import java.io.File; import java.io.FileOutputStream; import java.io.InputStream; import java.net.URL; import java.net.URLConnection; import java.util.List; import java.util.Objects; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; /** * Store results in files. * @author code4crafter@gmail.com * @since 0.1.0 */ public class SougouImgPipeline { private String extension = ".jpg"; private String path; private volatile AtomicInteger suc; private volatile AtomicInteger fails; public SougouImgPipeline() { setPath("E:/pipeline/sougou"); suc = new AtomicInteger(); fails = new AtomicInteger(); } public SougouImgPipeline(String path) { setPath(path); suc = new AtomicInteger(); fails = new AtomicInteger(); } public SougouImgPipeline(String path, String extension) { setPath(path); this.extension = extension; suc = new AtomicInteger(); fails = new AtomicInteger(); } public void setPath(String path) { this.path = path; } /** * 下载 * @param url * @param cate * @throws Exception */ private void downloadImg(String url, String cate, String name) throws Exception { String path = this.path + "/" + cate + "/"; File dir = new File(path); if (!dir.exists()) { // 目录不存在则创建目录 dir.mkdirs(); } String realExt = url.substring(url.lastIndexOf(".")); // 获取扩展名 String fileName = name + realExt; fileName = fileName.replace("-", ""); String filePath = path + fileName; File img = new File(filePath); if(img.exists()){ // 若文件之前已经下载过，则跳过 System.out.println(String.format("文件%s已存在本地目录",fileName)); return; } URLConnection con = new URL(url).openConnection(); con.setConnectTimeout(5000); con.setReadTimeout(5000); InputStream inputStream = con.getInputStream(); byte[] bs = new byte[1024]; File file = new File(filePath); FileOutputStream os = new FileOutputStream(file, true); // 开始读取 写入 int len; while ((len = inputStream.read(bs)) != -1) { os.write(bs, 0, len); } System.out.println("picUrl: " + url); System.out.println(String.format("正在下载第%s张图片", suc.getAndIncrement())); } /** * 单线程处理 * * @param data * @param word */ public void process(List<String> data, String word) { long start = System.currentTimeMillis(); for (String picUrl : data) { if (picUrl == null) continue; try { downloadImg(picUrl, word, picUrl); } catch (Exception e) { fails.incrementAndGet(); } } System.out.println("下载成功: " + suc.get()); System.out.println("下载失败: " + fails.get()); long end = System.currentTimeMillis(); System.out.println("耗时：" + (end - start) / 1000 + "秒"); } /** * 多线程处理 * * @param data * @param word */ public void processSync(List<String> data, String word) { long start = System.currentTimeMillis(); int count = 0; ExecutorService executorService = Executors.newCachedThreadPool(); // 创建缓存线程池 for (int i=0;i<data.size();i++) { String picUrl = data.get(i); if (picUrl == null) continue; String name = ""; if(i<10){ name="000"+i; }else if(i<100){ name="00"+i; }else if(i<1000){ name="0"+i; } String finalName = name; executorService.execute(() -> { try { downloadImg(picUrl, word, finalName); } catch (Exception e) { fails.incrementAndGet(); } }); count++; } executorService.shutdown(); try { if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) { // 超时的时候向线程池中所有的线程发出中断(interrupted)。 // executorService.shutdownNow(); } System.out.println("AwaitTermination Finished"); System.out.println("共有URL: "+data.size()); System.out.println("下载成功: " + suc); System.out.println("下载失败: " + fails); File dir = new File(this.path + "/" + word + "/"); int len = Objects.requireNonNull(dir.list()).length; System.out.println("当前共有文件： "+len); long end = System.currentTimeMillis(); System.out.println("耗时：" + (end - start) / 1000.0 + "秒"); } catch (InterruptedException e) { e.printStackTrace(); } } /** * 多线程分段处理 * * @param data * @param word * @param threadNum */ public void processSync2(List<String> data, final String word, int threadNum) { if (data.size() < threadNum) { process(data, word); } else { ExecutorService executorService = Executors.newCachedThreadPool(); int num = data.size() / threadNum; //每段要处理的数量 for (int i = 0; i < threadNum; i++) { int start = i * num; int end = (i + 1) * num; if (i == threadNum - 1) { end = data.size(); } final List<String> cutList = data.subList(start, end); executorService.execute(() -> process(cutList, word)); } executorService.shutdown(); } } } HttpClientUtils.java http请求工具类import org.apache.http.Header; import org.apache.http.HttpEntity; import org.apache.http.NameValuePair; import org.apache.http.client.entity.UrlEncodedFormEntity; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.client.methods.HttpPost; import org.apache.http.client.methods.HttpUriRequest; import org.apache.http.conn.ssl.SSLConnectionSocketFactory; import org.apache.http.conn.ssl.TrustStrategy; import org.apache.http.entity.StringEntity; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.message.BasicNameValuePair; import org.apache.http.ssl.SSLContextBuilder; import org.apache.http.util.EntityUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import javax.net.ssl.HostnameVerifier; import javax.net.ssl.SSLContext; import javax.net.ssl.SSLSession; import java.io.IOException; import java.io.UnsupportedEncodingException; import java.security.GeneralSecurityException; import java.security.cert.CertificateException; import java.security.cert.X509Certificate; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; /** * @author code4crafter@gmail.com * Date: 17/3/27 */ public abstract class HttpClientUtils { public static Map<String, List<String>> convertHeaders(Header[] headers) { Map<String, List<String>> results = new HashMap<String, List<String>>(); for (Header header : headers) { List<String> list = results.get(header.getName()); if (list == null) { list = new ArrayList<String>(); results.put(header.getName(), list); } list.add(header.getValue()); } return results; } /** * http的get请求 * @param url */ public static String get(String url) { return get(url, "UTF-8"); } public static Logger logger = LoggerFactory.getLogger(HttpClientUtils.class); /** * http的get请求 * @param url */ public static String get(String url, String charset) { HttpGet httpGet = new HttpGet(url); return executeRequest(httpGet, charset); } /** * http的get请求，增加异步请求头参数 * @param url */ public static String ajaxGet(String url) { return ajaxGet(url, "UTF-8"); } /** * http的get请求，增加异步请求头参数 * * @param url */ public static String ajaxGet(String url, String charset) { HttpGet httpGet = new HttpGet(url); httpGet.setHeader("X-Requested-With", "XMLHttpRequest"); return executeRequest(httpGet, charset); } /** * @param url * @return */ public static String ajaxGet(CloseableHttpClient httpclient, String url) { HttpGet httpGet = new HttpGet(url); httpGet.setHeader("X-Requested-With", "XMLHttpRequest"); return executeRequest(httpclient, httpGet, "UTF-8"); } /** * http的post请求，传递map格式参数 */ public static String post(String url, Map<String, String> dataMap) { return post(url, dataMap, "UTF-8"); } /** * http的post请求，传递map格式参数 */ public static String post(String url, Map<String, String> dataMap, String charset) { HttpPost httpPost = new HttpPost(url); try { if (dataMap != null) { List<NameValuePair> nvps = new ArrayList<NameValuePair>(); for (Map.Entry<String, String> entry : dataMap.entrySet()) { nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue())); } UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(nvps, charset); formEntity.setContentEncoding(charset); httpPost.setEntity(formEntity); } } catch (UnsupportedEncodingException e) { e.printStackTrace(); } return executeRequest(httpPost, charset); } /** * http的post请求，增加异步请求头参数，传递map格式参数 */ public static String ajaxPost(String url, Map<String, String> dataMap) { return ajaxPost(url, dataMap, "UTF-8"); } /** * http的post请求，增加异步请求头参数，传递map格式参数 */ public static String ajaxPost(String url, Map<String, String> dataMap, String charset) { HttpPost httpPost = new HttpPost(url); httpPost.setHeader("X-Requested-With", "XMLHttpRequest"); try { if (dataMap != null) { List<NameValuePair> nvps = new ArrayList<NameValuePair>(); for (Map.Entry<String, String> entry : dataMap.entrySet()) { nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue())); } UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(nvps, charset); formEntity.setContentEncoding(charset); httpPost.setEntity(formEntity); } } catch (UnsupportedEncodingException e) { e.printStackTrace(); } return executeRequest(httpPost, charset); } /** * http的post请求，增加异步请求头参数，传递json格式参数 */ public static String ajaxPostJson(String url, String jsonString) { return ajaxPostJson(url, jsonString, "UTF-8"); } /** * http的post请求，增加异步请求头参数，传递json格式参数 */ public static String ajaxPostJson(String url, String jsonString, String charset) { HttpPost httpPost = new HttpPost(url); httpPost.setHeader("X-Requested-With", "XMLHttpRequest"); StringEntity stringEntity = new StringEntity(jsonString, charset);// 解决中文乱码问题 stringEntity.setContentEncoding(charset); stringEntity.setContentType("application/json"); httpPost.setEntity(stringEntity); return executeRequest(httpPost, charset); } /** * 执行一个http请求，传递HttpGet或HttpPost参数 */ public static String executeRequest(HttpUriRequest httpRequest) { return executeRequest(httpRequest, "UTF-8"); } /** * 执行一个http请求，传递HttpGet或HttpPost参数 */ public static String executeRequest(HttpUriRequest httpRequest, String charset) { CloseableHttpClient httpclient; if ("https".equals(httpRequest.getURI().getScheme())) { httpclient = createSSLInsecureClient(); } else { httpclient = HttpClients.createDefault(); } String result = ""; try { try { CloseableHttpResponse response = httpclient.execute(httpRequest); HttpEntity entity = null; try { entity = response.getEntity(); result = EntityUtils.toString(entity, charset); } finally { EntityUtils.consume(entity); response.close(); } } finally { httpclient.close(); } } catch (IOException ex) { ex.printStackTrace(); } return result; } public static String executeRequest(CloseableHttpClient httpclient, HttpUriRequest httpRequest, String charset) { String result = ""; try { try { CloseableHttpResponse response = httpclient.execute(httpRequest); HttpEntity entity = null; try { entity = response.getEntity(); result = EntityUtils.toString(entity, charset); } finally { EntityUtils.consume(entity); response.close(); } } finally { httpclient.close(); } } catch (IOException ex) { ex.printStackTrace(); } return result; } /** * 创建 SSL连接 */ public static CloseableHttpClient createSSLInsecureClient() { try { SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(new TrustStrategy() { @Override public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException { return true; } }).build(); SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, new HostnameVerifier() { @Override public boolean verify(String hostname, SSLSession session) { return true; } }); return HttpClients.custom().setSSLSocketFactory(sslsf).build(); } catch (GeneralSecurityException ex) { throw new RuntimeException(ex); } } } 运行由于网络等原因，我们发现并不能全部下载成功，不过可以多次运行尝试，可以实现较高的下载成功率。666，厉害了。。