Git Product home page Git Product logo

Comments (2)

Chrisma-98 avatar Chrisma-98 commented on September 28, 2024

已自行解决问题,可爬取每个链接后及时储存数据。
解决方案:修改piplines.py中的【class WeibocrawlerPipeline(object)】如下

def process_item(self, item, spider):
    base_dir = '结果文件' 
    if not os.path.isdir(base_dir):
        os.makedirs(base_dir)
    file_path = base_dir + os.sep + 'Userinfo' + '.csv'
    if not os.path.isfile(file_path):
        is_first_write = 1
    else:
        is_first_write = 0
    if item:
        with open(file_path, 'a', encoding='utf-8-sig', newline='') as f:
            writer = csv.writer(f)
            if is_first_write:
                header = [
                    'id', 'nick_name', 'user_id', 'gender', 'brief_introduction', 'mblogs_num',
                    'follows_num', 'fans_num', 'vip_level', 'authentication', 'verified_type',  '_level'
                ]
                writer.writerow(header)
            writer.writerow(
                [item[key] for key in item.keys()])
    return item

其中header 需跟实际情况自行修改。
header = [
'id', 'nick_name', 'user_id', 'gender', 'brief_introduction', 'mblogs_num',
'follows_num', 'fans_num', 'vip_level', 'authentication', 'verified_type', '_level'
]
跟items.py中的【class UserItem(Item)】类内容保持一致。

from weibocrawler.

liliners avatar liliners commented on September 28, 2024

兄弟,请问一下你Scrapy的版本是1.5.1,还是用的更高版本?我运行的时候报错了,想确定一下是不是版本问题。

from weibocrawler.

Related Issues (16)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.