September Roundup
Welcome to September’s monthly roundup. I took a break in August - one of the perks of working for myself - and now am busy catching up with what’s happened in the AI world. Hence the shorter newsletter than usual as I haven’t yet got back into the regular paper-reading habit!
Next month I’ll be at CogX in London. Would love to meet you if you’re attending - let me know.
Stuff I read
Fine-tuning LLMs to 1.58bit: extreme quantization made easy
Find the post: https://arxiv.org/abs/2407.14679
How to make LLMs smaller? LLMs are large models, and the largest require dedicated GPU machines to run. Quantisation - reducing the precision of parameters - is a key way to reduce the size of LLMs so they can be run on cheaper hardware.
Enter BitNet. Earlier this year, Microsoft shared a paper where they used an extreme form of quantisation - having the model parameters be one of {-1, 0 or 1}. That's just 1.58bits per parameter, as compared to the 16 or 32 bits that are regularly used. But, the BitNet model was trained from scratch, limiting the usefulness of this quantisation to those who have the resources to train from scratch.
This month, Hugging Face showed how they took an existing LLM (Llama 3) and finetuned it to use the same 1.58 bit quantisation. The challenge is in balancing the loss in performance while doing so. The blog post goes into details of how they balanced this trade-off.
Women in tech groups 'can’t run on inspiration alone'
Find the article: https://www.bbc.co.uk/news/articles/c7858w2yj75o
Groups that support women can be fantastic for women in the tech industry. But, they’re struggling to survive, as this BBC article describes.
This is hot on the heels of a report last week showing that the percentage of women in US tech roles has hardly changed over the past 20 years.
We know that women have a harder time in this industry, and that difficulties are compounded when other factors like race, age or disability are also there. If you're working in a tech company and want to support women, consider long-term sustainable funding for one of the many groups or events out there.
Other links
Work with me
I work with organisations who are building AI - as a technical advisor, coach and speaker. Get in touch if you’d like to talk about working together.