> For the complete documentation index, see [llms.txt](https://ai.saikatkumardey.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai.saikatkumardey.com/from-rules-to-vectors-how-nlp-changed-over-time.md).

# From Rules to Vectors: How NLP Changed Over Time

This is a quick summary of Manning’s [article](https://www.amacad.org/sites/default/files/publication/downloads/Daedalus_Sp22_09_Manning.pdf) on the history and future of natural language processing and foundation models.

* Manning divides NLP history into four eras based on different approaches:
  * **Machine translation**: simple word-level translation with rules and lookups (1950-1969).
  * **Rule-based systems**: hand-built systems that handle syntax and reference with linguistic theories and knowledge-based AI (1970-1992).
  * **Empirical machine learning models**: statistical models that learn from data with labels or annotations for specific tasks or domains (1993-2012).
  * **Deep learning models**: neural network models that learn from raw data without labels or annotations and from self-supervision objectives for general tasks or domains (2013-present).
* LPLMs (Large Pretrained Language Models) are foundation models, which are models that can do many tasks after being trained on lots of data via self-supervision. They can also use other types of data, such as images, sounds, or actions.
* Foundation models have pros and cons for NLP and beyond:
  * Pros: high performance on many NLP tasks; learning from diverse and rich data sources; easy adaptation to different tasks or domains.
  * Cons: costly and slow to train; hard to understand and measure; not very reliable or flexible.
* Manning concludes that foundation models are an exciting and important direction for AI. He suggests that we need to learn more about how they work and how to improve them.