“Okaaay,” I said slowly. “So basically you need someone to constantly monitor the prices for different vacation locations and put the data together in a way that will be easy for your agents to offer your clients the best deals.”

“Exactly!” Shalom said animatedly, but then his face fell. “Except that it’s so mind-numbing that no one wants to do it.”

I smiled, happy I could help him. “Don’t worry about it. No human may want to do it, but the computer doesn’t mind. I can set up a program for you that’ll do the work for you.”

What Is Screen Scraping Technology?

Previously, we’ve discussed APIs, which allow two computer programs to transfer data between each other. But there are some cases when you want the data from a program or website, but you can’t use an API to extract it, either because the program doesn’t have an API, or you don’t have access to it.

So, what can you do when an API is not an option?

There’s the manual option, like Shalom tried, which involves hiring people to look through a website and write down the important information. But that option is expensive for the employer, because it means paying a salary for someone to do this, and the job is terribly boring for the employee hired to do it. Some people hire cheap labor overseas for this, but screen scraping, which we’ll discuss in a second, is still cheaper.

Screen scraping technology is software that interacts with a computer system like a regular user. Meaning, instead of accessing data using an API, which is a backend “pipe” that software uses to transfer data in a specific way, the screen scraper looks at the website or the software the same way a human would. In other words, the screen scraper can copy information from the visual elements of the screen.

This can work one of two ways. A screen scraper can work on an open website, like an airline’s website or an online store, for example. Those kinds of websites are open to the public, and anyone can just go to their website and see their products and their prices. So just like any human can look at the website, a screen scraper can as well.

The screen scraper can be set up to go to a website, extract specific information, and copy it to an Excel file, or to wherever you want that information to be saved. Then a human can look at the Excel file, which will only contain the relevant information, because the screen scraper already did the work of extracting it.

Alternately, the screen scraper can be directed to scrape a website or platform that requires a login. In that case, you can give the screen scraper your username and password, it will sign in for you, and then it can access and scrape the data on that website just like it does in the first scenario.

“You’re saying there’s a way to set up a screen scraper that will go through my whole list of travel sites and take out the relevant information for me the same way an employee would?” Shalom asked incredulously.

“That’s exactly what I’m saying. And the technology will do it faster, too.”

I smiled at Shalom’s expression. He looked like he found this too good to be true.

“How would it work?” he asked.

Setting up a Screen Scraper

There are many screen scrapers online that you can set up to use for your needs. Most of them are in the form of a Chrome or browser extension. You generally need to install the screen scraper, and then once it’s installed, you can set it up.

Some screen scrapers have an easy setup method: the screen scraper records your screen while you go through the step by step of opening up the website that you want, and clicking on the buttons to get to the data that you want. Once the screen scraper records you doing it once, it can then replicate what you did and do the same thing over and over so that your data is constantly updated.

If you need something a bit more complex, like scraping data on a website based on certain criteria or from a website that requires a login, there are special screen scrapers that can do that, but their setup is more complex. In those cases, it’s often worth getting a computer programmer or a software expert to set it up for you. Once set up, though, it can run on its own and is so simple to use.

“I’m sold!” Shalom sported a huge smile. “If I never need to hire for this position again, then this is definitely for me.”

(Originally featured in Mishpacha, Issue 1011)