In this issue:

• AI without NVidia
• Teaching tiny computers to do AI
• A Year with LLMs, real-world learnings

AI without NVidia

I know a few people who have invested in NVidia this year and are rather pleased with themselves. After all, it’s gone up 100% over the year and is a darling stock. I’m pleased for them, but in general, I’ve been telling them to take their money and be satisfied rather than hoping for further spectacular gains. There are a number of reasons that I don’t believe that NVidia will retain it’s eye watering valuation, and one of them came from a paper quietly announced on arxiv.org, which publishes pre-print academic papers.

While it’s pretty technical, the gist is that the authors have managed to make high performing LLM’s without using matrix multiplication, which is the mathematical operation that big AI models need to run on NVidia style hardware. Their work is still in the early stages, but it’s already producing high quality results using way less power and on potentially much less expensive hardware than NVidia. You can read the abstract and the whole paper here.

The sheer success of NVidia is spawning competition from a large variety of different sources, so expect more reports from me as they appear.

Teaching tiny computers to do AI

While huge LLM’s are making headlines in both positive and negative ways, AI is also reaching into devices that at a first glance should have no chance of running neural networks. Edge Impulse is a company that has created an easy-to-use platform that creates AI applications that can run on hardware that costs as little as $5. Now, these applications are not going to compete with ChatGPT anytime soon, but they will do things such as recognise gestures, identify objects, or classify audio on tiny, low powered devices.

This means that say, security cameras can do their own processing of video, and only send footage to the cloud that needs further investigation. Another example is that an industrial machine can self-monitor and raise an alarm if a part gets loose and starts making noises that are unfamiliar. This has a profound impact the cost of scaling big Internet of Things networks.

I’ve used Edge Impulse in a few fun projects already and it’s extremely impressive both in how simple it can make AI and how much intelligence can be squeezed out of cheap hardware.

A Year with LLM’s, real-world learnings

I recently came across a paper written by people who had spent the last year creating real-world LLM applications. It’s a long read, and is worth the time if you have it, but I came away with a few major bullet points.

• AI does not spell the end of programming. The big take-away from this artical is how much in-depth, technical and operational effort it takes to make a working, reliable AI application.
• Design to keep people in the loop. The authors strongly advise to plan for human feedback and control in AI systems. Basically, AI’s can assist people a lot, but can rarely perform completely reliably on their own, and may never do. LLMs will return output even when they shouldn’t and this has to be acknowledged and allowed for.
• Use the smallest model that will get the job done. It’s tempting to use GPT4 for everything, it can be prohibitively expensive to run at even modest levels. Smaller models are vastly cheaper to operate, and for many tasks perform at nearly the same level. They also respond much faster. This can translate to an offering that is actually financially viable as well as more responsive to the user, unlike many products today which are heavily loss making.

All-in-all the message is that making a good AI application is a serious business, requiring new marketing and operational skills as well as technical ones. Naive implementations can cost a lot, both in terms of money and reputational damage, as many companies are starting to find.

That’s it for this newletter, please share it with friends if you think it’s useful, tell me if it’s not.

Mike