2017-02-22 64 views
1

Я хочу очистить содержимое с веб-сайта http://www.jobs.ch. Результатом должен быть сценарий, где я могу указать срок работы, например. Бизнес-аналитик и получить все задания, которые есть в названии. Я предполагаю, что должен использовать многоступенчатый подход, собирая сначала все соответствующие ссылки с каждой страницы, сохраняя их, а затем извлекать описание задания.Скремблирование Javascript-веб-страницы с BeautifulSoup и Python

Возможно ли это, чтобы достичь этой цели? Или мне нужно использовать также селен, поскольку сайт находится в реакции.js?

Вот начало моего сценария:

from bs4 import BeautifulSoup 
import urllib2 

jobsFile = urllib2.urlopen("http://www.jobs.ch/en/vacancies/?term=business+analyst") 
jobsHtml = jobsFile.read() 
jobsFile.close() 

soup = BeautifulSoup(jobsHtml) 
jobsAll = soup.find_all("a") 
for links in soup.find_all('a'): 
    print (links.get('href')) 

Выход из консоли:

python jobplatform.py 
/Library/Python/2.7/site-packages/bs4/__init__.py:181: UserWarning: 
No parser was explicitly specified, so I'm using the best available 
HTML parser for this system ("lxml"). This usually isn't a problem, 
but if you run this code on another system, or in a different virtual 
environment, it may use a different parser and behave differently. 

The code that caused this warning is on line 8 of the file 
jobplatform.py. To get rid of this warning, change code that looks 
like this: 

BeautifulSoup([your markup]) 

to this: 

BeautifulSoup([your markup], "lxml") 

markup_type=markup_type)) 
None 
/en/ 
/en/login/ 
/en/register/ 
/en/vacancies/ 
/en/companies/ 
http://www.jobs.ch/en/sucheBerater.php 
http://www.jobs.ch/en/tipps 
http://www.jobs.ch/en/ecom/ 
/de/stellenangebote/?term=business 
/fr/offres-emplois/?term=business 
/en/vacancies/?term=business 
/en/vacancies/ 
None 
None 
None 
None 
/en/vacancies/?page=1&term=business&web-results=1 
None 
None 
/en/companies/79912-bayer-business-services-gmbh/ 
/en/vacancies/detail/7376115/?source=vacancy_search 
/en/companies/79912-bayer-business-services-gmbh/ 
/en/companies/48196-kotra-korea-business-center/ 
/en/vacancies/detail/7397077/?source=vacancy_search 
/en/companies/48196-kotra-korea-business-center/ 
/en/companies/66172-diwisa-distillerie-willisau-sa/ 
/en/vacancies/detail/7363589/?source=vacancy_search 
/en/companies/66172-diwisa-distillerie-willisau-sa/ 
/en/companies/2859-paul-scherrer-institut/ 
/en/vacancies/detail/7359642/?source=vacancy_search 
/en/companies/2859-paul-scherrer-institut/ 
/en/companies/49314-pit-offices-gmbh/ 
/en/vacancies/detail/7344672/?source=vacancy_search 
/en/companies/49314-pit-offices-gmbh/ 
/en/companies/27786-zuehlke-engineering-ag/ 
/en/vacancies/detail/7176356/?source=vacancy_search 
/en/companies/27786-zuehlke-engineering-ag/ 
/en/companies/1802-six-payment-services-ag/ 
/en/vacancies/detail/7396870/?source=vacancy_search 
/en/companies/1802-six-payment-services-ag/ 
/en/companies/49420-mettler-toledo-gruppe/ 
/en/vacancies/detail/7384998/?source=vacancy_search 
/en/companies/49420-mettler-toledo-gruppe/ 
/en/companies/16414-partners-group/ 
/en/vacancies/detail/7279253/?source=vacancy_search 
/en/companies/16414-partners-group/ 
/en/companies/4005-johnson-johnson/ 
/en/vacancies/detail/7397184/?source=vacancy_search 
/en/companies/4005-johnson-johnson/ 
/en/companies/44340-amgen/ 
/en/vacancies/detail/7359993/?source=vacancy_search 
/en/companies/44340-amgen/ 
/en/companies/1802-six-payment-services-ag/ 
/en/vacancies/detail/7357631/?source=vacancy_search 
/en/companies/1802-six-payment-services-ag/ 
/en/companies/16649-fritschi-unternehmensberatung-gmbh/ 
/en/vacancies/detail/7369054/?source=vacancy_search 
/en/companies/16649-fritschi-unternehmensberatung-gmbh/ 
/en/companies/19002-hays-schweiz-ag/ 
/en/vacancies/detail/7389632/?source=vacancy_search 
/en/companies/19002-hays-schweiz-ag/ 
/en/companies/5977-canon-schweiz-ag/ 
/en/vacancies/detail/7236919/?source=vacancy_search 
/en/companies/5977-canon-schweiz-ag/ 
/en/companies/40039-vorwerk-international-strecker-co/ 
/en/vacancies/detail/7374142/?source=vacancy_search 
/en/companies/40039-vorwerk-international-strecker-co/ 
/en/companies/2263-zuercher-kantonalbank/ 
/en/vacancies/detail/7299359/?source=vacancy_search 
/en/companies/2263-zuercher-kantonalbank/ 
/en/companies/10673-accenture/ 
/en/vacancies/detail/6664788/?source=vacancy_search 
/en/companies/10673-accenture/ 
/en/companies/38308-addexpert-gmbh/ 
/en/vacancies/detail/7386047/?source=vacancy_search 
/en/companies/38308-addexpert-gmbh/ 
/en/companies/1802-six-swiss-exchange-ag/ 
/en/vacancies/detail/7357633/?source=vacancy_search 
/en/companies/1802-six-swiss-exchange-ag/ 
/en/vacancies/?page=1&term=business 
/en/vacancies/?page=2&term=business 
/en/vacancies/?page=3&term=business 
/en/vacancies/?page=4&term=business 
/en/vacancies/?page=5&term=business 
/en/vacancies/?page=6&term=business 
/en/vacancies/?page=124&term=business 
/en/vacancies/?page=2&term=business 
None 
http://jobcloud.ch/c/en/about/ 
http://jobcloud.ch/c/en/about/ 
http://jobcloud.ch/c/en/about/team/ 
http://jobcloud.ch/c/en/we-are-jobcloud/ 
None 
http://www.jobs.ch/en/newest.php 
http://www.jobs.ch/en/info.php?info=agb 
http://www.jobs.ch/en/info.php?info=pp 
None 
http://jobcloud.ch/c/en/products/international-recruiting/ 
/en/ 
http://www.jobs.ch/en/sitemap.php 
http://jobcloud.ch/c/en/about/contact/ 
http://jobcloud.ch/ 
http://www.facebook.com/jobs.ch 
http://twitter.com/jobs_ch 
http://www.xing.com/company/jobcloudag 
http://www.youtube.com/jobspunktch 
http://plus.google.com/113239437813300663024/ 
http://www.flickr.com/photos/jobsag 
+2

Возможный дубликат [Web-соскоб странице JavaScript с Python] (http://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python) –

ответ

2

Как указано в @Teemu Risikko комментарий, вы можете использовать dryscrape или селен. Вот решение с использованием dryscrape:

from bs4 import BeautifulSoup 
import dryscrape 

my_url = "http://www.jobs.ch/en/vacancies/?term=business+analyst" 
session = dryscrape.Session() 
session.visit(my_url) 
response = session.body() 
soup = BeautifulSoup(response) 
jobsAll = soup.find_all("a") 
for links in soup.find_all('a'): 
    print (links.get('href')) 

Решение довольно просто с dryscrape но установка пакета может быть сложно (используйте кварты < = 55) ...