Myujth

Set-up

I have the next-page button element from this page,

<li class="Pagination-item Pagination-item--next Pagination-item--nextSolo ">
 <button type="button" class="Pagination-link js-veza-stranica kist-FauxAnchor" data-page="2" data-href="https://www.njuskalo.hr/prodaja-kuca?page=2" role="link">Sljedeća&nbsp;<span aria-hidden="true" role="presentation">»</span></button>
 </li>

I need to obtain the url in the data-href attribute.

Code

Using the following simple xpath to the button element in scrapy shell,

response.xpath('//*[@id="form_browse_detailed_search"]/div/div[1]/div[5]/div[1]/nav/ul/li[8]/button').extract_first() 

I retrieve,

'<button type="button" class="Pagination-link js-veza-stranica" data-page="2">Sljedećaxa0<span aria-hidden="true" role="presentation">»</span></button>'

Question

Where did the data-href attribute go to?

How do I obtain the url?

The data-href attribute is most likely being calculated by some JavaScript code running in your browser. If you look at the raw source code of this page ("view source code" option in your browser), you won't find that attribute there.

The output you see on developer tools is the DOM rendered by your browser, so you can expect differences between your browser view and what Scrapy actually fetches (which is the raw HTML source). Keep in mind that Scrapy doesn't execute any JavaScript code.

Anyway, a way to solve this would be building the pagination URL based on the data-page attribute:

from w3lib.url import add_or_replace_parameter
...

next_page = response.css('.Pagination-item--nextSolo button::attr(data-page)').get()
next_page_url = add_or_replace_parameter(response.url, 'page', next_page)

w3lib is an open source library: https://github.com/scrapy/w3lib