Happy to share that my student, José Ángel Bello Pérez, successfully defended his Bachelor’s Thesis on the design and implementation of an automated system for collecting and extracting information from online data marketplaces. He gave an excellent presentation and achieved an outstanding grade of 9.5/10. Big congratulations!
The developed system combines flexible, modular web crawling techniques to build a repository of over 12k commercial data products from 2.4k+ providers in a NoSQL database. It leverages Large Language Models (LLMs) to flexibly process the downloaded information and extract 80+ homogeneous metadata fields. Finally, a statistical and quantitative analysis of this structured dataset was carried out to identify patterns, trends, and relevant characteristics of the current supply in commercial data marketplaces.
Watch the demo video here.
Beyond the results of the analysis, the system stands out for its modularity and flexibility. As a result, we have managed to:
- Reduce onboarding time: Significantly faster integration of information from new marketplaces or domains.
- Enable historical reprocessing: The ability to reprocess web pages collected in previous years to identify long-term trends or to add new information.
- Adapt effortlessly: Easily modify prompt programs to extract different information fields.
Many thanks to my previous students, Miguel Eleno and Álvaro Pérez, who laid the foundation for this work. Another brick in the Data Economy Observatory at UPM. I am very much looking forward to welcoming new students to help build this system that will decisively contribute to increasing the transparency of emerging data markets.
Santiago Andrés Azcoitia 
Deja un comentario