Social media: Сutting-edge big data solution for an emerging network

According to the Digital 2023 April Global Statshot Report, the global count of active social media users has surged to 4.8 billion, approaching an impressive milestone of nearly 60% of the world's population.

Each user engages with an average of 6.6 networking platforms, with at least 25% browsing to find inspiration for things to do and products to buy, particularly among younger generations. No wonder all networks nowadays take it upon themselves to recommend something that will keep their users constantly engaged, interested, and invested.

Here’s where recommender systems based on big data take center stage. And where our Client, a rising social network company competing with giants like Facebook, X (former Twitter), and TikTok, approached us in 2021 with their project.

SENLA became the development center for this product. This means that a large team of our top experts provides an end-to-end solution to the Client, developing the product from the ground up and offering full-cycle support.

Our Client came to our big data experts with a few key inquiries: 

Advertising random products to random users yielded little results. Our Client aimed to target potential consumers more specifically, improve the user experience through personalized suggestions, introduce discoveries, and narrow down choices.

It was clear to SENLA’s big data engineers that what the Client needed was to implement a recommendation system.

A recommendation system (or recommender), typically powered by artificial intelligence (AI) and machine learning (ML), is a big data engine that makes predictions and recommendations based on user preferences.

In today's digital landscape, the vast array of online products and services makes recommender systems indispensable. They enhance customer satisfaction by streamlining searches and presenting new findings. This, in turn, boosts platform engagement and drives sales growth.

Recommender systems employ different AI and ML algorithms, come in different flavors, and can process a variety of factors such as past purchases, search history, likes, shares, image or text analysis, and any other user activity.

Choosing suitable recommendation algorithms for your product is a significant task in itself. Different models are used to tackle different challenges of linking users to resources and information that would’ve otherwise remained undiscovered.

For the initial product launch, our experts and the Client's product owners opted for Amazon Personalize, a ready-made recommender system. As more data is collected from people using the network, SENLA will develop and implement a custom-made solution with superior quality and project-specific algorithms.

But first, to prepare datasets and start training the recommendation system, the data must be transferred to a data storage platform.

Following the selection of the Agile development methodology (with 2-week sprints), establishing communication channels, and scheduling work meetings and demos (daily meetups, sprint retrospectives), we started the work by choosing the data pipeline and data storage solutions.

Data pipelines are processes and tools that move raw data from sources, such as applications and databases, to destinations, such as data warehouses and data lakes. Once imported, the data becomes available for reporting, analysis, and extracting actionable business insights.

To meet all specifications, SENLA’s engineers proposed building the recommender system using a data lake and an ELT pipeline.

A data lake serves as a central storage hub that houses an extensive volume of data in its original, raw format.

This solution is preferred over a data warehouse due to its ability to store nearly all data types, be it text, multimedia, likes, user profiles, and more, which is usually the case with a social network.

Raw unstructured data is crucial for preparing datasets to train the recommendation AI & ML algorithms. Additionally, data lakes offer comprehensive history, allowing for in-depth analytics and reporting.

An ELT pipeline is the process of extracting (E) data from a source, loading (L) it into a destination with no intermediate processing, and then transforming (T) it within the repository for further consumption.

ELT is typically employed with data lakes and in ML use cases instead of the more traditional ETL (where data is transformed before loading) because ELT accommodates larger volumes and a wider variety of data types.

Such pipelines are better suited to transfer data to a destination platform for training and deploying ML models, which is the main task of this project. Besides, ELT allows data scientists to ingest data in its original form, and then flexibly transform it to extract insights on an as-needed basis while using familiar tools.

The project's entire ecosystem relies on Amazon’s AWS infrastructure, with data sourced from two separate databases to handle its heterogeneity and varied structure.

Within the lake, three separate layers together with cataloging instruments perform various data transformations:

A CDC (Change Data Capture) process tracks updated/new data from the sources, later to be extracted and loaded to the lake. In this way, all data is stored with complete “history”, which is essential for the recommender system to learn from and enables long-term analysis with full dynamics over time.

The recommendation system itself consists of two parts: the first component orchestrates dataset preparation and the creation/tuning of the recommendation engine. The second part coordinates the system’s re-training events and actions.

The recommender system produces sets of recommended entities tailored to specific users. These results are stored in the system and subsequently delivered to consumers on a schedule after double-checking their relevance.

Our recommender’s primary goal is to encourage and nurture strong engagement of users with the Client's network.

The system empowers tracking of emerging trends, studies publication lifecycles, and delivers precise suggestions to customers for enhanced monetization and customer satisfaction.

In the upcoming phases of the project:

This will be the focus of future successful software development by SENLA for our Client. Reach out if you need assistance with your big data project.