如何在多次重定向到抓取站点后登录？

link之家

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

如何在多次重定向到抓取站点后登录？

import scrapy
from scrapy.http import Request
class DownloadFirmwareSpider(scrapy.Spider):
    name = "download_firmware"
    start_urls = ['https://support.fortinet.com/Download/FirmwareImages.aspx']
    def parse(self, response):
        yield scrapy.FormRequest.from_response(
            response,
            formid='login_form',
            formdata={
                'username': 'sample_username', 
                'password': 'sample_password',
            callback=self.after_login
    def after_login(self, response):
        #Specify pages to crawl here:
        pagelist = ['https://support.fortinet.com/Download/FirmwareImages.aspx']
        for page in pagelist:
            yield Request(url= page,
                callback=self.action)
    def action(self, response):
        # Continue processing each page...
        return

2018-08-20 13:39:49 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: fortigate)
2018-08-20 13:39:49 [scrapy.utils.log] INFO: Versions: lxml 4.2.4.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.5 |A
naconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2p  14 Aug 2018), cryptography 2.3.1, Platform Window
s-7-6.1.7601-SP1
2018-08-20 13:39:49 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'fortigate', 'NEWSPIDER_MODULE': 'fortigate.spiders', 'ROBOTSTXT_OBEY': True, 'SPID
ER_MODULES': ['fortigate.spiders']}
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled item pipelines:
2018-08-20 13:39:49 [scrapy.core.engine] INFO: Spider opened
2018-08-20 13:39:49 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-08-20 13:39:49 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-08-20 13:39:50 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://support.fortinet.com/robots.txt> (referer: None)
2018-08-20 13:39:50 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2
fFirmwareImages.aspx> from <GET https://support.fortinet.com/Download/FirmwareImages.aspx>
2018-08-20 13:39:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2fFirmwareImages.aspx> (ref
erer: None)
2018-08-20 13:39:50 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://support.fortinet.com> from <GET https://customersso1.fortin
et.com/robots.txt>
2018-08-20 13:39:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://support.fortinet.com> (referer: None)
2018-08-20 13:39:51 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://customersso1.fortinet.com/login/saml-idp-auth/?prefix=jrx0g
5n1etn0aoy9&next=%2Fsaml-idp%2Fjrx0g5n1etn0aoy9%2Flogin_proces%2F> from <POST https://customersso1.fortinet.com/saml-idp/jrx0g5n1etn0aoy9/login/>
2018-08-20 13:39:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://customersso1.fortinet.com/login/saml-idp-auth/?prefix=jrx0g5n1etn0aoy9&next=%2Fsaml-i
dp%2Fjrx0g5n1etn0aoy9%2Flogin_proces%2F> (referer: https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2fFirmwareImages.aspx)
2018-08-20 13:39:51 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2
fFirmwareImages.aspx> from <GET https://support.fortinet.com/Download/FirmwareImages.aspx>
2018-08-20 13:39:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2fFirmwareImages.aspx> (ref
erer: https://customersso1.fortinet.com/login/saml-idp-auth/?prefix=jrx0g5n1etn0aoy9&next=%2Fsaml-idp%2Fjrx0g5n1etn0aoy9%2Flogin_proces%2F)
2018-08-20 13:39:52 [scrapy.core.engine] INFO: Closing spider (finished)
2018-08-20 13:39:52 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 4961,
 'downloader/request_count': 9,
 'downloader/request_method_count/GET': 8,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 138281,
 'downloader/response_count': 9,
 'downloader/response_status_count/200': 4,
 'downloader/response_status_count/301': 1,
 'downloader/response_status_count/302': 3,
 'downloader/response_status_count/404': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 8, 20, 17, 39, 52, 169836),
 'log_count/DEBUG': 10,
 'log_count/INFO': 7,
 'request_depth_max': 2,
 'response_received_count': 5,