All Questions
2,752 questions
2
votes
0
answers
62
views
python-requests-html render inconsistent result
background:
by default the website is only showing few names and there s a "moreBtn" to generate the full list
code idea:
create Html session, render with script clicking the "moreBtn&...
- 31
1
vote
2
answers
51
views
Importing geographic data with WFS works on Chrome but not on Python
I am trying to pull a geojson file from here.
The JSON appears as expected when I paste that link into Chrome or Safari. However, I get the following error every time when I run the following code on ...
- 121
1
vote
2
answers
68
views
How to detect and scrape a specific language version of a multilingual publication, if available?
I wrote a python script for scraping data from WHO website, I wanted to retrieve Title, author name, date, pdf link and child page link from parent page (i applied some filters on parent page)
I am ...
- 11
0
votes
0
answers
52
views
Using the free-proxy library with requests to access general https websites
When basically requesting a proxy, what happens is that it delivers an http that currently seems to me to be unusable because the vast majority of sites use https and this causes the request to be ...
- 2,177
-1
votes
1
answer
33
views
Need help scraping FAA N Number database as I can't seem to communicate with the url
Trying to pull data from FAA N Number results but request.get() doesn't seem to be working.
I followed this tutorial (https://www.youtube.com/watch?v=QhD015WUMxE) and was able to scrape the website he ...
0
votes
3
answers
83
views
How to extract particular tags from soup using python?
From below webpages I like to extract data:
https://www.ams.usda.gov/services/enforcement/organic/settlements https://www.ams.usda.gov/services/enforcement/organic/settlements-2023
"03/19/2025&...
-1
votes
2
answers
59
views
bs4 cannot extract text from an element
import requests
from bs4 import BeautifulSoup
url = 'https://www.tori.fi/recommerce/forsale/item/22362242'
headers = {"User-Agent": "Mozilla/5.0"}
response = requests.get(url, ...
- 9
1
vote
1
answer
97
views
Failed to parse the total results from a webpage, of which my existing script can parse one-third
I've created a script that issues a POST HTTP request with the appropriate parameters to fetch the town, continent, country, and inner_link from this webpage. The script can parse 69 containers, but ...
- 149
0
votes
1
answer
110
views
Failed to identify the reason why my script is missing a few results while scraping a webpage
I've created a script in Python to scrape consultant links from this webpage based on the country filter United States, located in the left sidebar. The webpage shows 2,025 results. However, when I ...
- 164
1
vote
1
answer
92
views
How to scrape website which has hidden data inside table?
I am trying to Scrape Screener.in website to extract some information related to stocks.
However while trying to extract Quarterly Results section there are some field which is hidden and when click ...
- 155
0
votes
1
answer
59
views
How to create a BeautifulSoup variable that will allow find_all to identify HTML table
I am trying to create a single content variable based off of five website pages. The code I'm using is:
soup_a = []
for i in range(1,6):
url_a = f'https://www.mascotdb.com/native-american-high-...
- 27
0
votes
1
answer
70
views
Python Requests GET with Proxy - HTTPS scheme returns expected result but HTTP returns header
When setting any URL to use HTTPS as the scheme (i.e., https://), I get my desired response (i.e., page source), but any http url (i.e., http://) fails or I receive a header and I don't understand why ...
- 73
0
votes
2
answers
109
views
How to get data from public google sheet and more than 100 rows
im currently trying to scrape a google sheet. But I have serious troubles because I cant just use requests to get the page data because only the first 100 rows are loaded. So I tried to use selenium ...
0
votes
2
answers
69
views
Scraping dynamic data table with no easy references
I'm trying to get the data from a simple table from the following website (https://bvmf.bmfbovespa.com.br/clube-de-investimento/clube-de-investimento.aspx?Idioma=pt-br). I was able to get the data ...
- 27
0
votes
1
answer
59
views
How to extract the value of the link tel from the internal web page using Beautifulsoup?
I'm trying to write a script that can collect information about phones and add it to a dataframe. I have such a dataset with customer ID. At the same time, the phone numbers are stored inside the web ...
