MetaAI tested: doesn't exactly prove its own existence, but free is free.

Image credit: Meta

Meta's new big language model, Llama 3, conceptually powers a new chatbot called “Meta AI” that the social media and advertising company has installed into more and more of its apps and interfaces. How does this model stack up against other general purpose conversational AIs? It tends to reorder a lot of web search results, and it doesn't excel at anything, but hey – the price is right.

You can currently access Meta AI for free on the web at Meta.ai, on Instagram, Facebook, WhatsApp and maybe a few other places if those aren't enough. It had been available before, but the release of Llama 3 and the new Imagine image generator (not to be confused with Google's Imagen) prompted Meta to be promoted as the first stop for the AI-curious. After all, you'll probably use it accidentally because they've replaced your search box with it!

Mark Zuckerberg even said that he expects MetaAI to be “the most used and best AI assistant in the world.” It is important to have goals.

A quick reminder about our “review” process: this is a very informal review of the model, not with synthetic criteria but just asking general questions that normal people might ask. We compare the results to our experience with other models, or simply to what you'd expect to get from one. It's the furthest thing from comprehensive, but it's something anyone can understand and replicate.

We're always changing and adjusting our approach, and sometimes we'll add something odd that we come across or exclude things that didn't really seem relevant. For example, this time, although it is our general policy not to try to test the media's ethnicity (that's another can of worms), my colleague Ivan noticed that the Imagine model showed bias towards Indian people. doing. We'll have that article soon (we may already have the meta).

Also, as a PSA at the outset, you should know that an apparent bug on Instagram prevented me from deleting sent questions. So I would avoid asking anything that you don't want to show up in your search history. Also, the web version didn't work for me in Firefox.

News and current events

First, I asked Meta AI what was going on between Israel and Iran. He responded with a short, bulleted list, helpfully including dates, though he cited only one CNN article. Like many other prompts I tried, it ends up with a link to Bing Search on the web interface and on Google Search in Instagram. I asked Meta, and a spokesperson said that these are primarily search promotion partnerships.

(The images in this post are for reference only and do not necessarily show the full answer.)

To check if Meta AI was somehow piggybacking on Bing's own AI model (which in turn borrows from Microsoft OpenAI), I clicked through and looked at the Copilot answer to the suggested question. It also had a bulleted list with almost the same information but better inline links and more references. Definitely different.

MetaAI's response was factual and up-to-date, if not particularly eloquent. The mobile response was quite a bit more compressed and difficult to source, so be aware that you're getting a shorter response there.

Next, I asked if there were any recent trends on TikTok that parents should be aware of. He responded with a high-level summary of what creators do on the social network, but nothing recent. Yes, I know people do “funny skits: humorous, relatable, or parody content” on TikTok, thanks.

Interestingly, when I asked a similar question about trends on Instagram, I got marketing-type phrases like “replying with rails creates conversation” and “AI creates new opportunities” and “ There was an enthusiastic response using text posts 'flourishing on the gram.' This is just a word-for-word, SEO Bait Instagram Trends Hootsuite post.

If I ask Meta's AI on Instagram about trends on Instagram, I'd expect something more interesting. If I want to read a chum, I'll just search for it.

History and context

I asked Meta AI to help me find some primary sources for some research I'm doing on late 19th century Supreme Court decisions.

His response relies heavily on an inoffensive but basic free SEO add-on post that lists several notable decisions from the 19th century. Not exactly what I asked for, and then at the end he also listed the 1896 founding document for the People's Party, a left-leaning party of the era. It doesn't really have anything to do with the Supreme Court, but MetaAI cited the page, which describes some of the justices as having anti-partisan views. A strange and irrelevant inclusion.

Other models provided context and a summary of the trends of the period. I would not use MetaAI as a research assistant.

Some basic trivia questions, such as who won the most medals at the 1984 Olympics and what notable events occurred that year, are answered and referenced extensively.

It's a bit annoying that it adds its reference numbers at the top and then the links at the bottom. What is the point of giving them numbers unless the numbers relate to some claim or fact? Some other models will reference inline, which is more convenient for research or fact-checking.

The conflict

I asked MetaAI why Donald Trump's supporters are predominantly old and white. This is the kind of question that is factual in a sense but obviously more sensitive than asking about the number of medals. The answer was pretty much the same, even pushing back the inherent claim of the question.

Unfortunately, he didn't provide any sources or links to find it. Too bad, because these kinds of conversations are a great opportunity for people to learn something new.

I also asked about the rise of white nationalism and got a solid list of reasons why we are seeing these things around the world. “Addressing these factors through education, compassion, and inclusive policies is critical to countering the rise of white nationalism and fostering a more equal society,” Meta AI said. So he didn't take the aggressively neutral stance that you sometimes see. No links or sources on it, either; I suspect they are currently avoiding citations on some topics, which I understand, but is this where citations are most needed?

medical

I told MetaAI that my (hypothetical) nine-year-old was getting a rash after eating cupcakes and asked what I should do. Interestingly, he wrote the entire response and then deleted it, “Sorry, I can't help you with this request right now,” and told me that I had blocked him from completing the response. Sir, no.

So I asked him again and he gave me a similar answer (which you see above), which includes perfectly reasonable and general advice for someone looking to manage a possible allergic reaction. It was probably one of those “oops, maybe I shouldn't have said that” type rollbacks where the model only realizes what she's done too late.

The same goes for a question about supplements: He answered even-handedly and reasonably well, including questions about common dosages, costs and efficacy.

In mental health, her advice on anxiety and medication was as straightforward and safe as possible (basically, “I'm not a doctor, consult a specialist”), and when I asked if I had severe anxiety Who Should I Call, the National Suicide Prevention Lifeline (1-800-273-TALK) and the Crisis Text Line (741741), among other things. No links or sources, though. A reasonable and humane response.

Abstract

When I had the meta-AI summarize a recent article I wrote, it picked only key-sounding sentences to repeat from it, mostly verbatim. It's not exactly wrong, but I wouldn't say it's a summary.

And when he tweaked a sentence, the meaning changed slightly: I originally wrote that Intuitive Machines “will almost certainly be firing on all cylinders to land that multibillion-dollar deal.” can.” The summary states that the company will “almost certainly pursue what may be…” which may mislead some into thinking that IM is a company that will try for the deal. will In fairness, I could have stated it more clearly myself.

When I asked him to do the same thing in fewer words, he did better – sort of.

The synopsis was more concise, but it included claims by Musk that Starship could return samples from Mars within five years — in some specifics. did not Add to my article because he makes many such unsubstantiated claims. AI must have gotten it from the other four articles he cited, for some reason, in my “summary”. If I ask for an abstract, I don't expect a model to fetch external information to include.

Content

The marketing copy suggestions for a fictional clothing brand that I claim to promote are exactly what you'd expect, which shows how far the job goes. There is no doubt that Meta in particular has an endless supply of such topics for training. Pity the marketers — their toil of writing millions of pathetic pitches for their brands has dictated a style in which AI has become quite adept.

When asked for some jokes from Farmer, he gave some real stinkers:

Why did the farmer's chicken go to the vet?

Because it had chicken pox!

And here's another one:

Why did the farmer take his pig to the movie theater?

To see “Baby”!

Wow – scary. But we can't expect much better from these models. Mostly this type of question is just to see if it does something weird or even repeats something from a particular community — I'm not looking for content (for now).

Result

Meta has put its AI as the first layer for casual questions, and it works. But for the most part it looks like you're looking for what you ask about and cites freely from the top results. And half the time there was a search at the end anyway. So why not just use Google or Bing in the first place?

Some of the “suggested” questions I tried, such as tips for overcoming writer's block, produced results that didn't directly cite anyone (or the source). But they were also completely unreal. Again, a typical internet search no Powered by a huge language model within a social media app, accomplishes more or less the same thing with less craft.

Meta AI generated very straightforward, almost minimal responses. I don't necessarily expect AI to go beyond the scope of my original query, and in some cases that would be a bad thing. But when I ask what ingredients are needed for a recipe, isn't the goal of communicating with an AI that understands my intent and literally scrapes the list from the top Bing results? Does it offer anything else?

I'm not a big user of these platforms to begin with, but Meta AI hasn't convinced me that it's particularly useful for anything. To be fair it is one of the few models that are both free and stay up to date with current events by searching online. Comparing to the free Copilot model now and then on Bing, the latter generally worked better, but I hit my daily “conversation limit” after only a few conversions. (It's unclear if Meta will impose any usage limits on AI.)

If you can't be bothered to open a browser to search for “lunar new year” or “quinoa water ratio,” you can probably ask MetaAI if you're already in one of the company's apps. (And often, you are). You can't ask that from TikTok! Yet.