In the era of generative AI, where RAG (Retrieval-Augmented Generation) systems combine search models with large language models (LLM), the protection of personal data takes on particular importance. The key challenge is the identification of sensitive information (PII) in compliance with GDPR (General Data Protection Regulation), while maintaining operational efficiency. Is it always worth reaching for the most advanced techniques? How do different approaches impact processing time and costs? Let’s dive into this topic by answering questions that concern many decision-makers.
Is NER the best option?
Named Entity Recognition (NER) is one of the most advanced tools for identifying personal data. Based on natural language processing techniques such as BERT or GPT, it allows the detection of PII even in complex contexts. For example, in the sentence “Jan Kowalski requests contact at jan.kowalski@example.com,” NER will easily recognize the first name, last name, and email address.
However, technical sophistication comes at a price. NER models operating based on large LLMs, such as GPT-4, are costly to maintain and to process data. Their implementation may require hundreds of hours of work, and each query generates additional computational costs, particularly in large-scale systems. In real-time solutions, such as chatbots serving thousands of users daily, using LLMs can lead to delays and rising expenses. Therefore, NER works excellently in complex contexts, but for simpler applications, it is worth looking for alternatives.
Are regular expressions a relic of the past?
Regular expressions (regex) may seem outdated compared to NER, but their simplicity and efficiency remain invaluable. If we are searching for email addresses, phone numbers, or social security numbers (like PESEL), regex works instantly and does not burden the system. Example? Detecting a PESEL number in a document will take a fraction of a second, with almost zero operational costs.
Of course, regex has its limitations. When personal data appears in more complex linguistic forms, such as a description: “Please contact me regarding Jan Kowalski’s order,” this technique will not be sufficient. In such cases, support from more advanced methods is necessary. Regex therefore remains an ideal tool for handling repetitive and well-defined structures, but it will not replace solutions with greater flexibility.
How well do you know your data?
Business logic–based rules are an approach often overlooked but can be extremely effective. Based on the specific requirements of a given industry, systems can apply sets of rules that precisely identify personal data. Example? In banking systems, a customer number may be treated as sensitive data, which is accounted for in text processing rules.
The advantage of this approach is its speed and low cost. Business logic allows rules to be easily tailored to specific needs, eliminating the need to burden the system with complex AI models. However, for rules to work correctly, they must be continually updated in response to new data patterns. For many companies, this effort is worthwhile, especially when the goal is to reduce costs while maintaining GDPR compliance.
Is it possible to process data without revealing it?
Hashing and checksums are techniques that allow data to be processed without revealing its actual content. For example, the email address “jan.kowalski@example.com” can be converted into a unique hash, which cannot be reversed but can be compared with other hashes in the system. This solution is extremely secure and inexpensive to implement.
Unfortunately, hashing only works for data already known to the system. If new PII patterns appear, this technique becomes useless. However, combined with other methods, hashing can provide an additional layer of protection, especially in systems requiring high scalability and security.
Is a hybrid the answer to everything?
In practice, none of these methods is ideal on its own. That is why more and more companies are opting for a hybrid approach that combines different techniques to optimize performance. Regular expressions and business logic rules can be used as the first filter, eliminating data with simple structures. Then, more difficult cases can be forwarded for analysis with NER or LLM. Hashing can secure known personal data, minimizing their processing by more resource-intensive algorithms.
This strategy allows achieving a balance between accuracy, costs, and time. In systems that handle millions of queries, the hybrid approach can save both money and computing resources without compromising GDPR compliance.
What to choose for your business?
The decision to select techniques for recognizing personal data should depend on the specifics of the business and the scale of operations. Does your company process millions of queries daily? Then regex and business logic rules can be your allies. Are the data non-standard and difficult to predict? NER or LLM may be indispensable. Or perhaps you want to maximize the security of data already known to the system? Hashing will give you peace of mind.
Regardless of the choice, the key is a pragmatic approach—one that considers both business needs and technological limitations. After all, protecting personal data is not only a matter of legal compliance but also of building customer trust and ensuring operational efficiency.
