The week's pick
Random Articles
Reseach Article
Adaptive Encoding for Scalable Segment Storage in Advertising and Recommendation Systems
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 187 - Number 19 |
| Year of Publication: 2025 |
| Authors: Xingpeng Xiao, Pengfei Gao |
10.5120/ijca2025925275
|
Xingpeng Xiao, Pengfei Gao . Adaptive Encoding for Scalable Segment Storage in Advertising and Recommendation Systems. International Journal of Computer Applications. 187, 19 ( Jul 2025), 34-37. DOI=10.5120/ijca2025925275
Abstract
Effective segment storage is an integral infrastructure problem in recommendation systems and targeted advertising systems. These systems need to store and retrieve enormous numbers of user-segment relationships on-the-fly in real-time while keeping latency low and storage overhead minimal. Conventional segment storage solutions have high storage overhead and low query performance, particularly as the number of user accounts and segments grows. In this work, we introduce an adaptive storage system for segment stores that automatically chooses among storing in an array form versus bitmaps versus run-length encoding depending on each user's segment listing's sparsity or density. Through optimized threshold computation, the system can automatically choose the most space-effective storage mechanism for each user segment listing without having to compare all compression options. Our approach is shown through experimentation to save significant storage space while also optimizing segment retrieval operations at a low latency. The new methods are well-suited for large-scale real-time recommendation engines and targeted advertising systems as well as for large-scale streaming services. A preliminary version of this encoding model was published as a U.S. patent [1].
References
- X. Xiao, Y. Shi, et al., “Efficient Storage for Segment Store,” U.S. Patent US20240403919A1, Apr. 2024. [Online]. Available: https://patents.google.com/patent/US20240403919A1/en
- A. Agarwal, R. Agrawal, P. Jain, V. Kashyap, R. Khandekar, and M. Sabharwal, “Scaling personalized advertising on LinkedIn,” in Proc. 23rd ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD), 2017, pp. 1953–1961.
- B. Schwartz, E. Weinstein, and A. Roy-Chowdhury, “FLEDGE: Privacy-preserving interest-based advertising,” Google Ads Developer Blog, 2022. [Online]. Available: https://developer.chrome.com/en/docs/privacy-sandbox/fledge/
- K. Wu, E. Otoo, and A. Shoshani, “Compressing bitmap indexes for faster search operations,” In Proc. SSDBM, 2006, pp. 99–108.
- D. Lemire, O. Kaser, and K. Aouiche, “Compressed bitmap indexes: Beyond unions and intersections,” Software: Practice and Experience, vol. 40, no. 2, pp. 131–147, 2010.
- S. Chambi, D. Lemire, O. Kaser, and R. Godin, "Better bitmap performance with Roaring bitmaps," Software: Practice and Experience, vol. 46, no. 5, pp. 709–719, 2016.
- D. Lemire, L. Boytsov, and N. Kurz, "Roaring bitmaps: Implementation of an optimized compressed bitmap index," Software: Practice and Experience, vol. 48, no. 4, pp. 867–886, 2018.
- A. Colantonio and R. Di Pietro, “Concise: Compressed 'n' composable integer set,” Information Systems, vol. 38, no. 8, pp. 1084–1097, 2013.
- M. Zukowski, S. Heman, N. Nes, and P. Boncz, “Super-scalar RAM-CPU cache compression,” in Proc. IEEE Int. Conf. on Data Engineering (ICDE), 2006, pp. 59–70.
- D. Abadi, D. Carney, U. Çetintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik, “Aurora: a new model and architecture for data stream management,” The VLDB Journal, vol. 12, no. 2, pp. 120–139, 2003.
- P. Covington, J. Adams, and E. Sargin, “Deep neural networks for YouTube recommendations,” in Proceedings of the 10th ACM Conference on Recommender Systems (RecSys), 2016, pp. 191–198.
- A. Criminisi, J. Shotton, and E. Konukoglu, “Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning,” Foundations and Trends in Computer Graphics and Vision, vol. 7, no. 2–3, pp. 81–227, 2012.
Index Terms
Keywords