About Rafał
Rafał is an Author, software engineer, trainer, and consultant specializing in information retrieval. He helps companies throughout the entire software lifecycle—from requirements gathering and architecture to implementation, deployment, scaling, and tuning. In his free time, he is a novice carpenter and an ultra runner, with varying degrees of success.
In his own words: "I'm running a single man consulting company where I'm focused on helping customers with their search systems, architecture as well as helping translating the business needs to implementation plans. Part of the work I'm doing is also software engineering, engaged in various activities - i.e. electronic identity product development."
Where to Fond Rafał
Let’s start from the beginning - how did you get involved in the search tech industry?
I became interested in information retrieval at university, but I never had the chance to truly dive into it at that time. A few years later, my team and I were implementing an e-commerce platform for a Polish retailer, and we used Apache Lucene — this was the first spark, but I wouldn't call it a love at first sight. To me, at that time it felt kind strange, but was already fast and its language capabilities were already impressive. Two years later, we implemented Apache Solr for an e-commerce platform in Poland and began evangelizing Solr to our clients and potential customers. This is how it all began for me.
Tell us about your current role and what you’re working on these days.
I'm a software engineer helping push the products further and a consultant helping customers with search related problems, architecture and team management.
Could you describe a ‘favorite failure’ — a setback that ultimately led to an important lesson or breakthrough in your work?
I think my favorite failure was issues with customer data leading to the removal of older data for a customer. Luckily it was no longer needed, but it is really something that I still remember after 20 years since making it, and it allowed me to grow as a professional.
What are some of the biggest misconceptions about search that you often encounter?
One of the biggest ones that I still encounter is belief that search is a one time problem. Some companies still think that once you implement and deploy the first version of your search platform you are done and you don't need to do anything more, which as we know is not true.
How do you envision AI and machine learning impacting search relevance and data insights over the next 2-3 years?
We already see a lot of changes in the market and a shift towards automated, self learning systems - it is happening more and more. I think all the breakthroughs in AI, faster and better models and more processing power will allow processing more data, more signals, prepare data easier and finally lead to a better search.
Can you share an example of a particularly challenging production issue you’ve encountered in your work with search technologies, and the process you used to resolve it?
I remember one of the performance problems for a customer with a massive Elasticsearch cluster. They were running an older version of Elasticsearch back then and were not able to upgrade because of the need of data re-indexing. It was like running in circles. We had to stabilize the cluster that was in a bad shape for weeks or months at that time and was breaking on a daily basis. Observability was non existing and we had to start from the basics - introduce monitoring, introduce metrics to the application, learn where the pressure comes from, work with the team to not only optimize the search engine, but the application using it as well. A lot of work, but rewarding at the end.
Are there any open-source tools or projects beyond Elasticsearch and OpenSearch that have significantly influenced your work?
I think two of the most influential projects to me are Apache Lucene and Apache Solr. That's because of my interest in search and because in the beginning I've been doing a lot of work with both of them. Yes, I know it is not anything new, but I probably wouldn't be doing a lot of search work if it wasn't for those two.
Is there a log error/alert that terrifies/annoys you in particular?
Maybe not exactly terrifying, but anything that says that the production system is down and I know that there is no backup system that handles the traffic. I don't like when the users suffer from the service not being available.
What is a golden tip for optimizing search performance that you’ve picked up in your years of experience?
I think I've learn through the years that you can't fix what you can't measure. If you don't have visibility into your system metrics then performance optimization can be hard or even impossible, depending where the problem is. You can of course optimize the system just by using your experience, but having the visibility into your who system will help and make things way easier.
What is the most unexpected or unconventional way you’ve seen search technologies applied?
I've seen a near real time queue implemented on top of Apache Solr, just because it was the source of truth for the data and no one wanted another infrastructure element in the architecture.
If you're building something from scratch - what does your ideal search tech stack look like?
That depends on the use - case. For machine learning and data processing I use Python and the tooling it provides. For backend services that needs to be lower level I use Golang or Rust. For frontend - Svelte, React, and recently Vue.js. For anything apart from that I use JVM based stack - Java or Kotlin, Spring, Quarkus, Ktor and anything that allows me to do the job faster.
Give us a spicy take/controversial opinion on something related to Search
Well, not that controversial, but remember garbage in, garbage out - we tend to forget about it too much.
Can you suggest a lesser-known book, blog, or resource that would be valuable to others in the community?
I think there is a lot of materials that we can learn from, but I would point out to the blog of Vespa.ai I learned a lot of and I'm still learning a lot from the majority of their publications, worth checking out.
