Text Embedding for Fun and Profit | Grey Noise Blog

Summary: GreyNoise published a technical blog explaining text embedding, the process of converting text into numerical vectors so models can perform tasks like nearest-neighbor search or classification, and contrasting GPT models, which mask only the last word in a sequence, with BERT models, which mask random words bidirectionally. The post describes building a custom RoBERTa-based tokenizer and model from scratch on at least 100k samples of HTTP payloads that don't follow typical English syntax, requiring 9-10GB of GPU memory, and notes GreyNoise's in-development NoiseGPT feature that converts natural language into GNQL queries and Sift, which aims to reduce hundreds of thousands of daily HTTP payloads to roughly 15 new and relevant ones.
What you should know: GreyNoise is building its own embedding model because standard English-trained encoders don't fit the HTTP payloads it sees, and its Sift feature aims to cut hundreds of thousands of daily payloads down to about 15 - a reminder that the tooling deciding what your team sees is itself a choice, not a given.

Read the original at GreyNoise Blog ↗