Find all the span tags with the following attributes (class equal to green) = person names
from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen(‘http://www.pythonscraping.com/pages/warandpeace.html')
bs = BeautifulSoup(html, “html.parser”)
all the names are in span tag of class green, in order of appearance in the text
bs.find_all(tagName, tagAttributes)
attributes are dictionary
nameList = bs.findAll(‘span’, {‘class’: ‘green’})
for name in nameList:
print(name.get_text())
Anna
Pavlovna Scherer
Empress Marya
Fedorovna
Prince Vasili Kuragin
Anna Pavlovna
St. Petersburg
the prince
Anna Pavlovna
Anna Pavlovna
the prince
the prince
the prince
Prince Vasili
Anna Pavlovna
Anna Pavlovna
the prince
Wintzingerode
King of Prussia
le Vicomte de Mortemart
Montmorencys
Rohans
Abbe Morio
the Emperor
the prince
Prince Vasili
Dowager Empress Marya Fedorovna
the baron
Anna Pavlovna
the Empress
the Empress
Anna Pavlovna’s
Her Majesty
Baron
Funke
The prince
Anna
Pavlovna
the Empress
The prince
Anatole
the prince
The prince
Anna
Pavlovna
Anna Pavlovna
titles = bs.find_all([‘h1’, ‘h2’,‘h3’,‘h4’,‘h5’,‘h6’])
print([title.get_text() for title in titles])
[‘War and Peace’, ‘Chapter 1’]
red texts are dialogues
allText = bs.find_all(‘span’, {‘class’:{‘green’, ‘red’}})
print([text for text in allText])
[Well, Prince, so Genoa and Lucca are now just family estates of the
Buonapartes. But I warn you, if you don’t tell me that this means war,
if you still try to defend the infamies and horrors perpetrated by
that Antichrist- I really believe he is Antichrist- I will have
nothing more to do with you and you are no longer my friend, no longer
my ‘faithful slave,’ as you call yourself! But how do you do? I see
I have frightened you- sit down and tell me all the news., Anna
Pavlovna Scherer, Empress Marya
Fedorovna, Prince Vasili Kuragin, Anna Pavlovna, St. Petersburg, If you have nothing better to do, Count [or Prince], and if the
prospect of spending an evening with a poor invalid is not too
terrible, I shall be very charmed to see you tonight between 7 and 10-
Annette Scherer., Heavens! what a virulent attack!, the prince, Anna Pavlovna, First of all, dear friend, tell me how you are. Set your friend’s
mind at rest,, Can one be well while suffering morally? Can one be calm in times
like these if one has any feeling?, Anna Pavlovna, You are
staying the whole evening, I hope?, And the fete at the English ambassador’s? Today is Wednesday. I
must put in an appearance there,, the prince, My daughter is
coming for me to take me there., I thought today’s fete had been canceled. I confess all these
festivities and fireworks are becoming wearisome., If they had known that you wished it, the entertainment would
have been put off,, the prince, Don’t tease! Well, and what has been decided about Novosiltsev’s
dispatch? You know everything., What can one say about it?, the prince, What has been decided? They have decided that
Buonaparte has burnt his boats, and I believe that we are ready to
burn ours., Prince Vasili, Anna Pavlovna, Anna Pavlovna, Oh, don’t speak to me of Austria. Perhaps I don’t understand
things, but Austria never has wished, and does not wish, for war.
She is betraying us! Russia alone must save Europe. Our gracious
sovereign recognizes his high vocation and will be true to it. That is
the one thing I have faith in! Our good and wonderful sovereign has to
perform the noblest role on earth, and he is so virtuous and noble
that God will not forsake him. He will fulfill his vocation and
crush the hydra of revolution, which has become more terrible than
ever in the person of this murderer and villain! We alone must
avenge the blood of the just one…. Whom, I ask you, can we rely
on?… England with her commercial spirit will not and cannot
understand the Emperor Alexander’s loftiness of soul. She has
refused to evacuate Malta. She wanted to find, and still seeks, some
secret motive in our actions. What answer did Novosiltsev get? None.
The English have not understood and cannot understand the
self-abnegation of our Emperor who wants nothing for himself, but only
desires the good of mankind. And what have they promised? Nothing! And
what little they have promised they will not perform! Prussia has
always declared that Buonaparte is invincible, and that all Europe
is powerless before him…. And I don’t believe a word that Hardenburg
says, or Haugwitz either. This famous Prussian neutrality is just a
trap. I have faith only in God and the lofty destiny of our adored
monarch. He will save Europe!, I think,, the prince, that if you had been
sent instead of our dear Wintzingerode you would have captured the
King of Prussia’s consent by assault. You are so eloquent. Will you
give me a cup of tea?, Wintzingerode, King of Prussia, In a moment. A propos,, I am
expecting two very interesting men tonight, le Vicomte de Mortemart,
who is connected with the Montmorencys through the Rohans, one of
the best French families. He is one of the genuine emigres, the good
ones. And also the Abbe Morio. Do you know that profound thinker? He
has been received by the Emperor. Had you heard?, le Vicomte de Mortemart, Montmorencys, Rohans, Abbe Morio, the Emperor, I shall be delighted to meet them,, the prince, But tell me,, is it true that the Dowager Empress wants Baron Funke
to be appointed first secretary at Vienna? The baron by all accounts
is a poor creature., Prince Vasili, Dowager Empress Marya Fedorovna, the baron, Anna Pavlovna, the Empress, Baron Funke has been recommended to the Dowager Empress by her
sister,, the Empress, Anna Pavlovna’s, Her Majesty, Baron
Funke, The prince, Anna
Pavlovna, the Empress, Now about your family. Do you know that since your daughter came
out everyone has been enraptured by her? They say she is amazingly
beautiful., The prince, I often think,, I often think how unfairly sometimes the
joys of life are distributed. Why has fate given you two such splendid
children? I don’t speak of Anatole, your youngest. I don’t like
him,, Anatole, Two such charming children. And really you appreciate
them less than anyone, and so you don’t deserve to have them., I can’t help it,, the prince, Lavater would have said I
lack the bump of paternity., Don’t joke; I mean to have a serious talk with you. Do you know I
am dissatisfied with your younger son? Between ourselves, he was mentioned at Her
Majesty’s and you were pitied…., The prince, What would you have me do?, You know I did all
a father could for their education, and they have both turned out
fools. Hippolyte is at least a quiet fool, but Anatole is an active
one. That is the only difference between them., And why are children born to such men as you? If you were not a
father there would be nothing I could reproach you with,, Anna
Pavlovna, I am your faithful slave and to you alone I can confess that my
children are the bane of my life. It is the cross I have to bear. That
is how I explain it to myself. It can’t be helped!, Anna Pavlovna]
if you want to find the number of times “the prince” is surrounded by tags on the example page, textcontet= the prince
nameList = bs.find_all(text=‘the prince’)
print(len(nameList))
7
This returns the first tag with the word “text” in the class_ attribute and “title” in the id attribute
title = bs.findall(id=‘title’, class=‘text’)
print([text for text in title])
these two lines are identical
bs.find_all(id=‘text’)
bs.find_all(“, {‘id’:‘text’})
[]
[
“
Well, Prince, so Genoa and Lucca are now just family estates of the
Buonapartes. But I warn you, if you don’t tell me that this means war,
if you still try to defend the infamies and horrors perpetrated by
that Antichrist- I really believe he is Antichrist- I will have
nothing more to do with you and you are no longer my friend, no longer
my ‘faithful slave,’ as you call yourself! But how do you do? I see
I have frightened you- sit down and tell me all the news.“
It was in July, 1805, and the speaker was the well-known
Anna
Pavlovna Scherer, maid of honor and favorite of the
Empress Marya
Fedorovna. With these words she greeted
Prince Vasili Kuragin, a man
of high rank and importance, who was the first to arrive at her
reception.
Anna Pavlovna had had a cough for some days. She was, as
she said, suffering from la grippe; grippe being then a new word in
St. Petersburg, used only by the elite.
All her invitations without exception, written in French, and
delivered by a scarlet-liveried footman that morning, ran as follows:
“
If you have nothing better to do, Count [or Prince], and if the
prospect of spending an evening with a poor invalid is not too
terrible, I shall be very charmed to see you tonight between 7 and 10-
Annette Scherer.“
“
Heavens! what a virulent attack!” replied
the prince, not in the
least disconcerted by this reception. He had just entered, wearing
an embroidered court uniform, knee breeches, and shoes, and had
stars on his breast and a serene expression on his flat face. He spoke
in that refined French in which our grandfathers not only spoke but
thought, and with the gentle, patronizing intonation natural to a
man of importance who had grown old in society and at court. He went
up to
Anna Pavlovna, kissed her hand, presenting to her his bald,
scented, and shining head, and complacently seated himself on the
sofa.
“
First of all, dear friend, tell me how you are. Set your friend’s
mind at rest,” said he without altering his tone, beneath the
politeness and affected sympathy of which indifference and even
irony could be discerned.
“
Can one be well while suffering morally? Can one be calm in times
like these if one has any feeling?” said
Anna Pavlovna. “
You are
staying the whole evening, I hope?“
“
And the fete at the English ambassador’s? Today is Wednesday. I
must put in an appearance there,” said
the prince. “
My daughter is
coming for me to take me there.“
“
I thought today’s fete had been canceled. I confess all these
festivities and fireworks are becoming wearisome.“
“
If they had known that you wished it, the entertainment would
have been put off,” said
the prince, who, like a wound-up clock, by
force of habit said things he did not even wish to be believed.
“
Don’t tease! Well, and what has been decided about Novosiltsev’s
dispatch? You know everything.“
“
What can one say about it?” replied
the prince in a cold,
listless tone. “
What has been decided? They have decided that
Buonaparte has burnt his boats, and I believe that we are ready to
burn ours.“
Prince Vasili always spoke languidly, like an actor repeating a
stale part.
Anna Pavlovna Scherer on the contrary, despite her forty
years, overflowed with animation and impulsiveness. To be an
enthusiast had become her social vocation and, sometimes even when she
did not feel like it, she became enthusiastic in order not to
disappoint the expectations of those who knew her. The subdued smile
which, though it did not suit her faded features, always played
round her lips expressed, as in a spoiled child, a continual
consciousness of her charming defect, which she neither wished, nor
could, nor considered it necessary, to correct.
In the midst of a conversation on political matters
Anna Pavlovna
burst out:
“
Oh, don’t speak to me of Austria. Perhaps I don’t understand
things, but Austria never has wished, and does not wish, for war.
She is betraying us! Russia alone must save Europe. Our gracious
sovereign recognizes his high vocation and will be true to it. That is
the one thing I have faith in! Our good and wonderful sovereign has to
perform the noblest role on earth, and he is so virtuous and noble
that God will not forsake him. He will fulfill his vocation and
crush the hydra of revolution, which has become more terrible than
ever in the person of this murderer and villain! We alone must
avenge the blood of the just one…. Whom, I ask you, can we rely
on?… England with her commercial spirit will not and cannot
understand the Emperor Alexander’s loftiness of soul. She has
refused to evacuate Malta. She wanted to find, and still seeks, some
secret motive in our actions. What answer did Novosiltsev get? None.
The English have not understood and cannot understand the
self-abnegation of our Emperor who wants nothing for himself, but only
desires the good of mankind. And what have they promised? Nothing! And
what little they have promised they will not perform! Prussia has
always declared that Buonaparte is invincible, and that all Europe
is powerless before him…. And I don’t believe a word that Hardenburg
says, or Haugwitz either. This famous Prussian neutrality is just a
trap. I have faith only in God and the lofty destiny of our adored
monarch. He will save Europe!“
She suddenly paused, smiling at her own impetuosity.
“
I think,” said
the prince with a smile, “
that if you had been
sent instead of our dear Wintzingerode you would have captured the
King of Prussia’s consent by assault. You are so eloquent. Will you
give me a cup of tea?“
“
In a moment. A propos,” she added, becoming calm again, “
I am
expecting two very interesting men tonight, le Vicomte de Mortemart,
who is connected with the Montmorencys through the Rohans, one of
the best French families. He is one of the genuine emigres, the good
ones. And also the Abbe Morio. Do you know that profound thinker? He
has been received by the Emperor. Had you heard?“
“
I shall be delighted to meet them,” said
the prince. “
But tell me,“
he added with studied carelessness as if it had only just occurred
to him, though the question he was about to ask was the chief motive
of his visit, “
is it true that the Dowager Empress wants Baron Funke
to be appointed first secretary at Vienna? The baron by all accounts
is a poor creature.“
Prince Vasili wished to obtain this post for his son, but others
were trying through the
Dowager Empress Marya Fedorovna to secure it
for
the baron.
Anna Pavlovna almost closed her eyes to indicate that neither she
nor anyone else had a right to criticize what
the Empress desired or
was pleased with.
“
Baron Funke has been recommended to the Dowager Empress by her
sister,” was all she said, in a dry and mournful tone.
As she named
the Empress,
Anna Pavlovna’s face suddenly assumed an
expression of profound and sincere devotion and respect mingled with
sadness, and this occurred every time she mentioned her illustrious
patroness. She added that
Her Majesty had deigned to show
Baron
Funke, and again her face clouded over with sadness.
The prince was silent and looked indifferent. But, with the
womanly and courtierlike quickness and tact habitual to her,
Anna
Pavlovna wished both to rebuke him (for daring to speak he had done of
a man recommended to
the Empress) and at the same time to console him,
so she said:
“
Now about your family. Do you know that since your daughter came
out everyone has been enraptured by her? They say she is amazingly
beautiful.“
The prince bowed to signify his respect and gratitude.
“
I often think,” she continued after a short pause, drawing nearer
to the prince and smiling amiably at him as if to show that
political and social topics were ended and the time had come for
intimate conversation- “
I often think how unfairly sometimes the
joys of life are distributed. Why has fate given you two such splendid
children? I don’t speak of Anatole, your youngest. I don’t like
him,” she added in a tone admitting of no rejoinder and raising her
eyebrows. “
Two such charming children. And really you appreciate
them less than anyone, and so you don’t deserve to have them.“
And she smiled her ecstatic smile.
“
I can’t help it,” said
the prince. “
Lavater would have said I
lack the bump of paternity.“
“
Don’t joke; I mean to have a serious talk with you. Do you know I
am dissatisfied with your younger son? Between ourselves” (and her
face assumed its melancholy expression), “
he was mentioned at Her
Majesty’s and you were pitied….“
The prince answered nothing, but she looked at him significantly,
awaiting a reply. He frowned.
“
What would you have me do?” he said at last. “
You know I did all
a father could for their education, and they have both turned out
fools. Hippolyte is at least a quiet fool, but Anatole is an active
one. That is the only difference between them.” He said this smiling
in a way more natural and animated than usual, so that the wrinkles
round his mouth very clearly revealed something unexpectedly coarse
and unpleasant.
“
And why are children born to such men as you? If you were not a
father there would be nothing I could reproach you with,” said
Anna
Pavlovna, looking up pensively.
“
I am your faithful slave and to you alone I can confess that my
children are the bane of my life. It is the cross I have to bear. That
is how I explain it to myself. It can’t be helped!“
He said no more, but expressed his resignation to cruel fate by a
gesture.
Anna Pavlovna meditated.
]
Difference between children and descendants
see the source code of the website to understand the structure
html = urlopen(‘http://www.pythonscraping.com/pages/page3.html')
bs = BeautifulSoup(html, ‘html.parser’)
This code prints the list of product rows in the giftList table, including the initial row of column labels.
children are always exactly one tag below a parent, whereas descendants can be at any level in the tree below a parent.
for child in bs.find(‘table’,{‘id’:‘giftList’}).children:
print(child)
|
Item Title
|
Description
|
Cost
|
Image
|
|---|
|
Vegetable Basket
|
This vegetable basket is the perfect gift for your health conscious (or overweight) friends!
Now with super-colorful bell peppers!
|
$15.00
|
|
|
Russian Nesting Dolls
|
Hand-painted by trained monkeys, these exquisite dolls are priceless! And by “priceless,” we mean “extremely expensive”! 8 entire dolls per set! Octuple the presents!
|
$10,000.52
|
|
|
Fish Painting
|
If something seems fishy about this painting, it’s because it’s a fish! Also hand-painted by trained monkeys!
|
$10,005.00
|
|
|
Dead Parrot
|
This is an ex-parrot! Or maybe he’s only resting?
|
$0.50
|
|
|
Mystery Box
|
If you love surprises, this mystery box is for you! Do not place on light-colored surfaces. May cause oil staining. Keep your friends guessing!
|
$1.50
|
|
Pay attention to NEXT siblings
html = urlopen(‘
http://www.pythonscraping.com/pages/page3.html')
bs = BeautifulSoup(html, ‘html.parser’)
for sibling in bs.find(‘table’, {‘id’:‘giftList’}).tr.next_siblings:
print(sibling)
|
Vegetable Basket
|
This vegetable basket is the perfect gift for your health conscious (or overweight) friends!
Now with super-colorful bell peppers!
|
$15.00
|
|
|
Russian Nesting Dolls
|
Hand-painted by trained monkeys, these exquisite dolls are priceless! And by “priceless,” we mean “extremely expensive”! 8 entire dolls per set! Octuple the presents!
|
$10,000.52
|
|
|
Fish Painting
|
If something seems fishy about this painting, it’s because it’s a fish! Also hand-painted by trained monkeys!
|
$10,005.00
|
|
|
Dead Parrot
|
This is an ex-parrot! Or maybe he’s only resting?
|
$0.50
|
|
|
Mystery Box
|
If you love surprises, this mystery box is for you! Do not place on light-colored surfaces. May cause oil staining. Keep your friends guessing!
|
$1.50
|
|
Find all the image and find it’s parent (td in this case)
navigate in dom tree
html = urlopen(‘http://www.pythonscraping.com/pages/page3.html')
bs = BeautifulSoup(html, ‘html.parser’)
print(bs.find(‘img’,
{‘src’:‘../img/gifts/img1.jpg’})
.parent.previous_sibling.get_text())
$15.00
Use regular expressions to find all the images in the websites
import re
html = urlopen(‘http://www.pythonscraping.com/pages/page3.html')
bs = BeautifulSoup(html, ‘html.parser’)
using regular expressions
images = bs.find_all(‘img’, {‘src’:re.compile(’..\/img\/gifts/img.*.jpg’)})
for image in images:
print(image[‘src’])
note that images is an object of beautiful soup
type(images)
../img/gifts/img1.jpg
../img/gifts/img2.jpg
../img/gifts/img3.jpg
../img/gifts/img4.jpg
../img/gifts/img6.jpg
bs4.element.ResultSet
Use lambda function to implement some nice functions
using anonymous functions
bs.find_all(lambda tag: len(tag.attrs) == 2)
[
,
|
Vegetable Basket
|
This vegetable basket is the perfect gift for your health conscious (or overweight) friends!
Now with super-colorful bell peppers!
|
$15.00
|
|
,
|
Russian Nesting Dolls
|
Hand-painted by trained monkeys, these exquisite dolls are priceless! And by “priceless,” we mean “extremely expensive”! 8 entire dolls per set! Octuple the presents!
|
$10,000.52
|
|
,
|
Fish Painting
|
If something seems fishy about this painting, it’s because it’s a fish! Also hand-painted by trained monkeys!
|
$10,005.00
|
|
,
|
Dead Parrot
|
This is an ex-parrot! Or maybe he’s only resting?
|
$0.50
|
|
,
|
Mystery Box
|
If you love suprises, this mystery box is for you! Do not place on light-colored surfaces. May cause oil staining. Keep your friends guessing!
|
$1.50
|
|
]
use of lambda functions to replace some built-in bs functions
bs.find_all(lambda tag: tag.get_text() == ‘Or maybe he\’s only resting?‘)
bs.find_all(“, text=‘Or maybe he\’s only resting?‘)
[“Or maybe he’s only resting?”]
bs.find_all(lambda tag: tag.get_text() == ‘Or maybe he\’s only resting?‘)
[Or maybe he’s only resting?]
