首页 > > 网络编程 > 其它 >

简单的爬虫，爬取博客园文章标题

2018-06-17 23:02:52来源：未知阅读 ()

花了几个小时写了一个小程序，没什么技术含量，第一次写博客。本人是个菜鸟，想记录一下自己的学习。

运行效果如图：

代码如下：

            string url = "https://www.cnblogs.com/";
            int pagNum = 10;
            string html = "";
            //string pater = "(<a class=\"titlelnk\" href=\"[\\s\\S]*\" target =\"_blank\">)([\\s\\S]*)(</a>)";
            string pater = "<a class=\"titlelnk\" href=\"(.*?)\" target=\"_blank\">(.*?)</a>";
            Regex regex = new Regex(pater);

            int num = 0;
            for (int i = 1; i < pagNum; i++)
            {
           string  url2 = url + "#p" + i;
                try
                {
                    WebRequest request = WebRequest.Create(url2);
                    Stream stream = request.GetResponse().GetResponseStream();
                    using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
                    {
                        html = reader.ReadToEnd();
                    }
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.ToString());
                    Console.ReadKey();
                }
              
                if (!string.IsNullOrEmpty(html))
                {
                    //  Match match = Regex.Match(html, pater);
                    foreach (Match ma in regex.Matches(html))
                    {
                        Match math = Regex.Match(ma.Value,pater);
                        string title = math.Groups[2].Value;
                        num++;
                        Console.WriteLine("第" + num + "个标题:\b" +title);
                        File.AppendAllText(@"d:\1.txt",title+ "\r\n");
                    }
                   
                }
            }
            Console.ReadKey();