Using GPT-4 to generate synthetic children’s stories to train tiny language models.

Link. ChatGPT training its replacements. Many workers can sympathize.

“researchers settled on a training data set containing roughly 2 million stories … networks with fewer layers but more neurons per layer were better at answering questions that required factual knowledge; conversely, networks with more layers and fewer neurons per layer were better at keeping track of characters and plot points”