What is a scraper, and why is there no graphical interface?

As a Lead Product Designer at IndieBI, I was part of the Product Trio, working with our Product Manager and Tech Lead to apply the concepts of continuous discovery to our work.

Responsibilities

UX Design · UI Design · User Research · Prototyping · Usability testing · Cross-functional collaboration · Feedback integration · Documentation

Client

IndieBI is a data platform for game developers and publishers, with a focus on PC and console sales analytics. It collects sales data from stores such as Steam, PlayStation, and Microsoft and displays it in easy-to-read charts. IndieBI identifies the best sales times and values while also providing insights to maximize revenue.

Imagine assembling IKEA furniture with four different instruction manuals, one of which is in Morse code. That is how complicated our user onboarding process used to be. Users had to navigate four different applications, with the main data collection tool being a command-line app. Self-onboarding was not an option; users needed our help at every stage, which was not ideal.

Four apps were required to complete the setup:

1. Steam - a desktop application for downloading our Scrapers tool.
2. Scrapers - a command-line tool that scrapes data.
3. Web application - for organizing scraped stock units into products.
4. Dashboards - a desktop application for displaying and analyzing data with charts.

IndieBI wants to grow and scale

• Manual onboarding created a significant barrier to growth and scalability.
• Maintenance of multiple applications was inefficient and expensive.
• Users churned faster due to difficulties in maintaining daily data downloads.

There is no direct competition

IndieBI stands out in the game analytics market because it collects precise sales data from major platforms such as Steam, PlayStation, and Microsoft. While Sparkers provides a comparable comprehensive solution, IndieBI's nuanced analytics outperform platforms such as Steam Spy, SteamDB, and the single-platform Steam Data Suite in terms of accuracy and depth.

We wanted to simplify the onboarding process without requiring assistance from our business team.

Merging four apps into one

We had to combine four separate apps into a single IndieBI desktop app. Here's what we had to do:

1. We needed to incorporate the Scrapers app's logic into the desktop app.
2. We needed to move the functionalities from the Web app to the desktop app.
3. We needed to embed the Dashboards app in the desktop app.
4. We no longer needed the Steam app because we would have our installation file.

While integrating points 2–4 was relatively simple, incorporating the Scrapers functionality was far more difficult, and this was my primary goal.

Metrics of success

• Boost the onboarding process.
• Enable clients to scrape data without having to refresh sessions or update credentials more frequently than every two weeks.
• Increase customer satisfaction by better understanding processing states and wait times.

This was definitely not a linear journey. Design processes frequently look like spaghetti rather than straight lines. To describe the entire process I went through for this project would take a novel, so I will keep it short.

The image below provides a rough idea of my design process.

To better understand our users and avoid technical bias, I decided to use the product as it was at that time. I went through the entire onboarding process with the help of our Product Manager. I must admit that it was even more confusing than I expected. I made a list of observations, which grew even more later.

‍Navigating business goals and workflow planning

We started with a series of meetings with our stakeholders and all necessary teams to understand business goals, collect existing knowledge, and build consensus among all teams on how to manage their current workflow and plan for future work. Project documentation was extensive because backend work started much earlier. I began by reviewing the documentation of scraper behaviors and discussing potential user stories with the team members. The entire documentation was a living organism because the situation changed over time as we discovered new insights.

We set the research goals, selected the users from our target, and scheduled user interviews to gather more information. Our findings from our user insights table and interviews revealed that key user frustrations were:

• The onboarding process was tedious, requiring the configuration of four different apps. Users have no idea how to navigate between all of these apps.
• Users reported issues with the command-line app, such as how to correctly configure scrapers or even paste their password.
• Users do not understand scraper states, and they are unsure whether their data is up to date.
• Users are unsure how long they will have to wait to see the first data on their dashboard.

Visualizing user touchpoints in app configuration

With the information we gathered about our users, we identified all of the touchpoints required to introduce the new app configuration. We then created a user journey map to help visualize these touchpoints. We used our main user persona to help us empathize with our users.

After determining the project requirements and user needs, I turned my attention to the technical details. While understanding scraper behaviors and store configurations presented challenges, I also needed to grasp how store accounts are set up on our backend to support frontend configurations. Each platform, such as Steam, PlayStation, and Xbox, has its own unique setup and dependencies. I started by outlining user flows for each store and designing diagrams for the new scraper structure, regularly consulting with our tech lead. This understanding of the architecture was critical for moving forward with the project.

I faced challenges with multiple documentation files, inconsistent naming conventions, and understanding scraper configurations across different stores. While some stores, like Steam, were relatively straightforward despite having multiple scrapers, others, such as Microsoft, posed significant challenges for both developers and myself. Our issues extended beyond complex configurations and lengthy wait times.

To address these challenges, I created a single source of truth—a comprehensive table consolidating all the necessary information for designing scraper configurations. I then analyzed the existing scraper states and developed a new structure. This new approach was tested with our users and developers. I also collaborated with a copywriter to establish clear and consistent naming conventions.

I collaborated closely with our development and business teams throughout the design process. We had regular design review sessions to discuss and improve my design concepts. These designs evolved over time into a functional prototype, which was tested to ensure its effectiveness and user satisfaction.

I created a prototype in Figma, wrote up a user testing plan, and then consolidated everything in the Maze app for the initial internal testing. I chose colleagues from various departments to ensure fresh perspectives. This internal testing helped me spot minor usability issues and prioritize more critical issues for user testing.

After refining the prototype based on this feedback, we began user testing in three phases, totaling seventeen sessions. Each round of testing provided valuable insights, which I used to enhance the prototype for the next group of users. This iterative approach led to the introduction of features like the 'Fix it' button, a revamped state architecture, and improved account configuration.

• Instead of four separate apps, IndieBI offers a single desktop application for both Mac and Windows. Users can forget about the command-line app and configure all stores in the new app.
• Unlocked growth: users can now configure the app without our assistance.
• "The 'Fix it' button provides an easy solution for most issues. It redirects the user to the source account and opens the exact location where they can resolve the problem.
• Users can now scrape data without requiring manual intervention, only interacting with the scrapers once every two weeks, in accordance with our maintenance standard.
• The new scraper states have significantly improved status/problem communication. There is no doubt that the data is fresh and that the scraper is functioning properly.
• The total scraping progress indicator gives users an idea of how long they will wait to see the first data.