| Software Savvy |

Software Savvy: Part 16

“Don’t worry about it. No human may want to do it, but the computer doesn’t mind. I can set up a program for you that’ll do the work for you.”

Shalom sighed as he waved me into his office. Before I sat down, he gestured to the petite young woman who had just walked out the door.

“She just quit on me. Been here less than a month. And I can’t keep anyone in this job for too long.”

My expression may have alerted him that I thought he sounded like a pretty bad boss, and he quickly clarified.

“The job is so tedious, but vital to my business’s success. I’m already paying way above the going rate, and still, I can’t get anyone to stay.”

Shalom is the CEO of a boutique travel agency. He’d invited me to his office to discuss issues he’d been dealing with, hoping that software might be the solution.

“What exactly is this job that no one wants to do?” I asked him. “How horrible could it be, already?”

He shrugged. “Pretty horrible, to be honest. I wouldn’t be willing to do it myself either, and I totally understand the people who keep quitting on me. It’s just frustrating because I really need someone to do it.” He looked dejected.

“This is how my business works: Our clients reach out to us because they want to go on the perfect vacation, and they want everything to be taken care of for them. We make our money by taking a fee for getting them incredibly good deals on flights, hotel stays, rental cars, and other attractions, plus an additional fee if we get them deals that are well below what they would have paid otherwise.”

“I hear.” I nodded, envisioning the kind of vacations they put together.

“So we need to constantly be on top of airfares and hotel prices and look out for deals on attractions so we can nab deals for our clients without compromising their standards,” he continued.

I was starting to see where he was going, but I let him get there himself. “Okay, I’m with you.”

“The miserable job that must be done is to take a list of websites — a very long list, I’ll admit — and check them twice a day to monitor prices. Then the data needs to be saved in a spreadsheet so our travel agents can compare prices without needing to actually look on all of the sites.”

“Okaaay,” I said slowly. “So basically you need someone to constantly monitor the prices for different vacation locations and put the data together in a way that will be easy for your agents to offer your clients the best deals.”

“Exactly!” Shalom said animatedly, but then his face fell. “Except that it’s so mind-numbing that no one wants to do it.”

I smiled, happy I could help him. “Don’t worry about it. No human may want to do it, but the computer doesn’t mind. I can set up a program for you that’ll do the work for you.”

What Is Screen Scraping Technology?

Previously, we’ve discussed APIs, which allow two computer programs to transfer data between each other. But there are some cases when you want the data from a program or website, but you can’t use an API to extract it, either because the program doesn’t have an API, or you don’t have access to it.

So, what can you do when an API is not an option?

There’s the manual option, like Shalom tried, which involves hiring people to look through a website and write down the important information. But that option is expensive for the employer, because it means paying a salary for someone to do this, and the job is terribly boring for the employee hired to do it. Some people hire cheap labor overseas for this, but screen scraping, which we’ll discuss in a second, is still cheaper.

Screen scraping technology is software that interacts with a computer system like a regular user. Meaning, instead of accessing data using an API, which is a backend “pipe” that software uses to transfer data in a specific way, the screen scraper looks at the website or the software the same way a human would. In other words, the screen scraper can copy information from the visual elements of the screen.

This can work one of two ways. A screen scraper can work on an open website, like an airline’s website or an online store, for example. Those kinds of websites are open to the public, and anyone can just go to their website and see their products and their prices. So just like any human can look at the website, a screen scraper can as well.

The screen scraper can be set up to go to a website, extract specific information, and copy it to an Excel file, or to wherever you want that information to be saved. Then a human can look at the Excel file, which will only contain the relevant information, because the screen scraper already did the work of extracting it.

Alternately, the screen scraper can be directed to scrape a website or platform that requires a login. In that case, you can give the screen scraper your username and password, it will sign in for you, and then it can access and scrape the data on that website just like it does in the first scenario.

“You’re saying there’s a way to set up a screen scraper that will go through my whole list of travel sites and take out the relevant information for me the same way an employee would?” Shalom asked incredulously.

“That’s exactly what I’m saying. And the technology will do it faster, too.”

I smiled at Shalom’s expression. He looked like he found this too good to be true.

“How would it work?” he asked.

Setting up a Screen Scraper

There are many screen scrapers online that you can set up to use for your needs. Most of them are in the form of a Chrome or browser extension. You generally need to install the screen scraper, and then once it’s installed, you can set it up.

Some screen scrapers have an easy setup method: the screen scraper records your screen while you go through the step by step of opening up the website that you want, and clicking on the buttons to get to the data that you want. Once the screen scraper records you doing it once, it can then replicate what you did and do the same thing over and over so that your data is constantly updated.

If you need something a bit more complex, like scraping data on a website based on certain criteria or from a website that requires a login, there are special screen scrapers that can do that, but their setup is more complex. In those cases, it’s often worth getting a computer programmer or a software expert to set it up for you. Once set up, though, it can run on its own and is so simple to use.

“I’m sold!” Shalom sported a huge smile. “If I never need to hire for this position again, then this is definitely for me.”


(Originally featured in Mishpacha, Issue 1011)

Oops! We could not locate your form.