Since ChatGPT arrived in late 2022, massive language fashions (LLMs) have continued to boost the bar for what generative AI techniques can accomplish. For instance, GPT-3.5, which powered ChatGPT, had an accuracy of 85.5% on widespread sense reasoning information units, whereas GPT-4 in 2023 achieved round 95% accuracy on the identical information units. Whereas GPT-3.5 and GPT-4 primarily centered on textual content processing, GPT-4o — launched in Might of 2024 — is multi-modal, permitting it to deal with textual content, pictures, audio and video.
Regardless of the spectacular developments by the GPT household of fashions and different open-source massive language fashions, Gartner, in its hype cycle for synthetic intelligence in 2024, notes that “generative AI has handed the height of inflated expectations, though hype about it continues.” Some causes for disillusionment embody the excessive prices related to the GPT household of fashions, privateness and safety considerations concerning information, and points with mannequin transparency. Small language fashions with fewer parameters than these LLMs are one potential resolution to those challenges.
Smaller language fashions are simpler and less expensive to coach. Moreover, smaller fashions might be hosted on-premises, offering higher management over the shared information with these language fashions. One problem with smaller fashions is that they are usually much less correct than their bigger counterparts. To harness the strengths of smaller fashions whereas mitigating their weaknesses, enterprises are domain-specific small fashions, which have to be correct solely within the specialization and use circumstances they help. This area specialization might be enabled by taking a pre-trained small language mannequin and fine-tuning it with domain-specific information or utilizing immediate engineering for extra efficiency positive aspects.
