Star-Rating-Prediction
Predicting star ratings from customer reviews.
- reviews
- ≈ 14,000
- models compared
- 17
- weighted F1 (stacking)
- 0.725
Product tour video
Overview
Predicting the stars from a review's text, based on around 14,000 English-language Trustpilot reviews of the online retailer Temu, collected with a custom Python scraper. As a team of three, we wrote a technical spec and pivoted from regression to multi-class classification; only the text is used, no user or order data. A Streamlit interface shows sentiment plus initial and refined rating for any text, keeps an exportable log and lets you switch between a classic and a deep learning model.
Data and method
The data is heavily imbalanced: about 52% 1★, 29% 5★ and few ratings in between, so we measure weighted F1 rather than accuracy alone. Features: TF-IDF (5,000 terms) plus eleven simple text features such as length, punctuation and capitalisation.
We compared eleven classic models with scikit-learn – from decision tree to XGBoost, voting and stacking – and six Keras models: deep MLP, CNN, LSTM, CNN-LSTM, BiLSTM with attention and a transformer.
Results
The best classic model is stacking (weighted F1 0.725, accuracy 0.733); among the neural models the deep MLP with TF-IDF leads (AUC ≈ 0.88). The transformer clearly lags at this data size – more complexity isn't automatically better.
The middle classes stay hard (macro F1 0.45). An optional 20% correction on 3★ reduces the pull towards the middle; a colour-coded sample of 100 reviews shows openly where the model is off. Together, the views surface recurring supply-chain pain points such as late deliveries and refund issues.
Key takeaway: deep learning needs more compute and more data; classic models train and infer cheaper and faster and work well on smaller, well-prepared data. Choosing a model means balancing accuracy, explainability and operating cost. (We initially explored AliExpress and Wish too, then focused on Temu because of local compute limits.)