Selenium and VBA - traps for Young Players

User avatar
ChrisGreaves
PlutoniumLounger
Posts: 15498
Joined: 24 Jan 2010, 23:23
Location: brings.slot.perky

Selenium and VBA - traps for Young Players

Post by ChrisGreaves »

Spawned from this thread
I shall update this post/thread as I learn more.
SeleniumBasic v2.0.9.0 is a great idea; an interface between the programmer's platform and the browser world.
That said:-

(1) There are enough threads out there to confirm this: Selenium often does NOT flush out its temporary files. I find them in "C:\Users\Chris075\AppData\Local\Temp\Selenium". This morning I wondered why my daily backup took so long. Answer: deleting the contents of that folder with a RoboCopy /MIR from an empty folder.
So I did a short run of 80 files, that is, accessed 80 web pages and saved the text in 80 Word2003 documents of about 25KB each.
1,400,000,000
79
17,721,519
The Selenium folder was 1.4 Giga Bytes, for 80 web pages/document files, almost 18MB per web access.
A fix is said to be placing .Quit after .Close. I shall try that on my next run.

[later]
Untitled.png
The cost of one web page is 30MB.
I was single-stepping and after the .Close and .Quit these folders disappeared, so Hooray!

[one hour later]The Temp folder holds steady with about 40MB, some 200 pages processed in 60 minutes.
Cheers
Chris
You do not have the required permissions to view the files attached to this post.
An expensive day out: Wallet and Grimace

LisaGreen
5StarLounger
Posts: 964
Joined: 08 Nov 2012, 17:54

Re: Selenium and VBA - traps for Young Players

Post by LisaGreen »

Hmmm.. For some reason I've avoided selenium. I think it has to do with my code on other peoples computers. I see no reason why I should have to install yet another program to a computer that isn't mine. So I *tend* to work with only VBA. Having said that and in connection with web sites, I'm learning python so I can access you tube play lists by calling python from vba. This would seem to be a better option than trawling the DOM.

Lisa

User avatar
ChrisGreaves
PlutoniumLounger
Posts: 15498
Joined: 24 Jan 2010, 23:23
Location: brings.slot.perky

Re: Selenium and VBA - traps for Young Players

Post by ChrisGreaves »

LisaGreen wrote:
16 Oct 2021, 17:45
Hmmm.. For some reason I've avoided selenium. I think it has to do with my code on other peoples computers. I see no reason why I should have to install yet another program to a computer that isn't mine. So I *tend* to work with only VBA. Having said that and in connection with web sites, I'm learning python so I can access you tube play lists by calling python from vba. This would seem to be a better option than trawling the DOM.
Hello again old-timer! :grin:

I like Selenium because ... Some 20 years ago I had a colleague promoted/used/sold "Act!" and had a few clients who wanted to migrate data in and out of the Act! database from different applications (Excel, dBaseIII, Outlook and so on)

I smelt money, and proposed that I write a set of routines that would migrate CSV,,Excel, dBaseIII, and Outlook formats to Act!, and migrate Act! to CSV, Excel, dBaseIII, and Outlook. I defined a single common data format. Then I set about writing eight sets of routines - four to convert INTO my common format, and four to convert OUT of my common format. Eight routines total. I argued that this was cheaper (less time for me, so more money per unit-time for me) that writing sixteen routines, one for every one of the 4x4 combinations of Input and Output.

Selenium is a similar idea, Selenium is a common interface, a membrane if you will, between our VBA/Python/C code and the Chrome/Firefox/ie/Edge browsers.

Cheers
Chris
An expensive day out: Wallet and Grimace

LisaGreen
5StarLounger
Posts: 964
Joined: 08 Nov 2012, 17:54

Re: Selenium and VBA - traps for Young Players

Post by LisaGreen »

hmmmmm number 2!

Didn't they say something similar about .NET!?

Lisa

User avatar
ChrisGreaves
PlutoniumLounger
Posts: 15498
Joined: 24 Jan 2010, 23:23
Location: brings.slot.perky

Re: Selenium and VBA - traps for Young Players

Post by ChrisGreaves »

LisaGreen wrote:
16 Oct 2021, 19:09
Didn't they say something similar about .NET!?
yep! And I remember someone pushing third-generation languages.
But who uses FORTRAN II nowadays? (grin)
Cheers
Chris
An expensive day out: Wallet and Grimace

User avatar
ChrisGreaves
PlutoniumLounger
Posts: 15498
Joined: 24 Jan 2010, 23:23
Location: brings.slot.perky

Re: Selenium and VBA - traps for Young Players

Post by ChrisGreaves »

ChrisGreaves wrote:
16 Oct 2021, 12:53
I shall update this post/thread as I learn more.
I have uploaded www.chrisgreaves.com/Downloads/20211017_0737.zip which I claim is enough Word2003/VBA code to get you started if you have ever wanted to grab data from the web and drop it into a DOCument for processing.

Please watch the video at WiseOwl. Do what it says about installing Selenium and the Chrome browser driver, then you are good to go.

Yahoo! seemed to know very little about my test subject "Chris Greaves Toronto Bonavista", but the searches did run and did produce five documents.

Cheers
Chris
An expensive day out: Wallet and Grimace

LisaGreen
5StarLounger
Posts: 964
Joined: 08 Nov 2012, 17:54

Re: Selenium and VBA - traps for Young Players

Post by LisaGreen »

Talking about web pages..
The requested page could not be found.

LOL!!
Though I suppose it's not funny really. hehehe.

Lisa

User avatar
HansV
Administrator
Posts: 78236
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Selenium and VBA - traps for Young Players

Post by HansV »

I have corrected the link. It should now work.
Best wishes,
Hans

LisaGreen
5StarLounger
Posts: 964
Joined: 08 Nov 2012, 17:54

Re: Selenium and VBA - traps for Young Players

Post by LisaGreen »

Hi Hans!

It opens a new tab and immediately closes it again. Though I could be missing somat.. No file I'm afraid.

Lisa

User avatar
ChrisGreaves
PlutoniumLounger
Posts: 15498
Joined: 24 Jan 2010, 23:23
Location: brings.slot.perky

Re: Selenium and VBA - traps for Young Players

Post by ChrisGreaves »

LisaGreen wrote:
17 Oct 2021, 12:59
It opens a new tab and immediately closes it again. Though I could be missing somat.. No file I'm afraid.
THANK YOU Hans. (I'm not sure what went wrong. I did check the link - I always make it in the URL box and make sure the d/l window pops up ...)

Lisa: This is what I see, a little box asking me where to save the file; I tried it just now with Hans's correction in place.
If you have some sort of auto-download switch turned on, the file is probably sitting on your hard drive, drumming its little fingers on your desk impatiently (grin)
Untitled.png
Cheers
Chris
You do not have the required permissions to view the files attached to this post.
An expensive day out: Wallet and Grimace

LisaGreen
5StarLounger
Posts: 964
Joined: 08 Nov 2012, 17:54

Re: Selenium and VBA - traps for Young Players

Post by LisaGreen »

Ok guys.. Got it. But I had to open a new tab and use the link there. Just a heads up.
Opening it now even!


Lisa

User avatar
HansV
Administrator
Posts: 78236
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Selenium and VBA - traps for Young Players

Post by HansV »

@Chris: it was probably a glitch of the forum software.
It prefixed your URL with that of the Lounge, resulting in an invalid link.
Best wishes,
Hans

User avatar
ChrisGreaves
PlutoniumLounger
Posts: 15498
Joined: 24 Jan 2010, 23:23
Location: brings.slot.perky

Re: Selenium and VBA - traps for Young Players

Post by ChrisGreaves »

HansV wrote:
17 Oct 2021, 13:43
@Chris: it was probably a glitch of the forum software.
It prefixed your URL with that of the Lounge, resulting in an invalid link.
HANS!
Do not attempt to break my current record of 55 years with computers: It is always my fault when something doesn't work!

Grudgingly
Chris
An expensive day out: Wallet and Grimace

User avatar
ChrisGreaves
PlutoniumLounger
Posts: 15498
Joined: 24 Jan 2010, 23:23
Location: brings.slot.perky

Re: Selenium and VBA - traps for Young Players

Post by ChrisGreaves »

OK, not a trap, but an exercise if you want to know more about using Selenium and the WiseOwl videos to do something useful:
I am on a small pension, so each month, a week before the pension comes in, I check the balances on all my accounts and make sure everything is reconciled so that on pension day, I can shuffle the money with equanimity, to where it is needed. This involves many keyboards and clicks to obtain data from the eight accounts for SB, Loan, EQ, TG, BMOC, BMOM, SBVisa, POCKet change, but to me it is worthwhile. I learn that my Net Worth is rising again.

As an example of the boring and repetitive tasks I am forced to perform robot-like, take my savings account:

The EQ savings account requires me to:-

Code: Select all

(1) Load the browser (FireFox)
(2) Go to the web site https://secure.eqbank.ca/login 
(3) Optionally re-type my user name ("")
(4) Log in with my password (“”)
(5) Make sure “Remember my Email” is checked ON
(6) Make sure Remember this device, I trust it and use it often. is checked ON
(7) Optionally answer the skill-testing question (first street lived on, first movie seen, first mother's maiden name ...)
(8) Locate “Welcome back, Christopher Total $3,683.58” and report the balance
(9) Log Out of the web page (optionally NOT closing the browser)
(10) repeat this exercise for the other accounts.
Note that this involves:

Code: Select all

(a) Parsing the web page to determine what data should be keyed in next (optional pages requesting confirmation of movies, streets, schools etc)
(b) Logging in with a password
(c) Detecting when I am at a page with a balance
(d) Obtaining that balance
(e) Copying that balance amount to the clipboard for use once I exit the browser.
If I were presenting a class in the use of Selenium, this would make a pretty good exercise to wake everybody up after lunch.

And yes, of course, destroy the program code and all backups immediately after this works; don't want anyone able to use this code to access your account, do we?
Alternatively, supply your VBA code with the passwords through the clipboard which, of course, you pre-load with the data before calling the macro to do its job.
The balances likewise could be assembled in the outgoing clipboard.

Cheers
Chris
An expensive day out: Wallet and Grimace