Quantcast
Channel: User Misha Krul - Stack Overflow
Viewing all articles
Browse latest Browse all 42

can't extract data using scrapy

$
0
0

I'm trying to pull an address name from the following page: https://property.spatialest.com/nc/durham/#/property/100016

property_spider.py:

from scrapy import Spiderfrom scrapy.selector import Selectorfrom property.items import PropertyItemclass PropertySpider(Spider):    name = "property"    allowed_domains = ["property.spatialest.com"]    start_urls = [ "http://property.spatialest.com/nc/durham/#/property/100016"    ]       def parse(self, response):        address = Selector(response).xpath("//html/body/main/div/div[2]/div/div[1]/div[2]/div/section/div/div[1]/div[2]/header/div/div/div[1]/div[2]/span")        address_item = PropertyItem()        address_item['address'] = address.xpath('span[@class="value "]/text()').extract()        yield address_item

The spider returns {'address': []} every time. I think maybe there is something wrong with the way I'm telling it to extract the data..

UPDATE:

It looks like it's not pulling in any data because the request gets cut off at the '#'

RESPONSE: <200 https://property.spatialest.com/nc/durham/>2019-03-16 13:59:12 [scrapy.core.scraper] DEBUG: Scraped from <200 https://property.spatialest.com/nc/durham/>{'address': []}```

Viewing all articles
Browse latest Browse all 42

Trending Articles