Generative AI for Farming – O'Reilly

AI is often seen as an enterprise for, by and for the wealthy. We are going to take a look at one. Digital GreenOf Peasant. Chat, an innovative AI bot designed to help small-scale farmers in developing countries access critical agricultural information. Developing countries have often implemented technological solutions that would never have occurred to engineers in rich countries. They solve real problems rather than appeal to venture capitalists to “let's start another Facebook.” Farmer.Chat is one of those solutions.

Farmer.Chat helps Agricultural Extension Agents (EAs) and farmers get answers to agriculture-related queries. It has been deployed in India, Ethiopia, Nigeria and Kenya. Although it was originally designed for EAs, farmers are increasingly using it directly. They are already used to asking questions online using social media. Providing online access to better, more reliable agricultural information quickly and efficiently was an obvious goal.

An AI application for farmers and EAs faces many hurdles. One of the biggest hurdles is location. Farming is hyperlocal. Two farms may be a mile apart, but if one is on a hill and the other in a valley, their soil, drainage, and perhaps climatic conditions will be completely different. Different microclimates, insects, crops: what works for your neighbor may not work for you.

There is data to answer hyperlocal questions about topics like fertilization and pest management, but it is spread across many databases with many owners: governments, NGOs, and corporations, etc. What works besides local information. Farmer.Chat uses all of these means to answer questions—but in doing so it must respect the rights of farmers and database owners. Farmers have a right to privacy; They don't want to share information about their farm or tell others what problems they are facing. Corporations may want to limit what data they disclose and how it is disclosed. Digital Green solves this problem. Farm stackA secure open source protocol for opt-in data sharing. End-to-end encryption is used for all connections. All data sources, including farmers and government agencies, choose what data they want to share and how to share it. They may decide to share certain types of data and not others, or they may impose restrictions on the use of their data (for example, restricting it to certain geographic areas). While nice opt-ins sound imposing, treating its data providers and its users with respect has allowed Farmer.Chat to build a trusted ecosystem for data sharing. In turn, that ecosystem leads to successful farms.

FormStack also enables secret expressions. Was data from the data provider used successfully? Did a farmer provide local knowledge that helped others? Or did they have problems with information? Data is always a two-way street. It's not just about using data, it's about optimizing it.

Translation is the most challenging issue for Digital Green and Farmer.Chat. Farmer.Chat currently supports six languages ​​(English, Hindi, Telugu, Amharic, Swahili, and Hausa) and Digital Green is working to add more. To serve EAs and farmers well, Farmer.Chat must also be multimodal — voice, text, and video — and reach farmers in their native languages. Although useful information is available in many languages, finding this information and answering a question in the farmer's language through voice chat is a formidable challenge. Farmer.Chat uses Google Translate, Azure, Whisper, and Bhashini (an Indian company that provides text-to-speech and other services for Indian languages), but there are still gaps. Even within a language, the same word can mean different things to different people. Many farmers measure their yield in bags of rice, but what is a “bag of rice”? This means 10 kg to a farmer, and 5 kg to any other buyer or seller. This is one area where keeping the extension agent in the loop is critical. An EA will be aware of issues such as local usage, local colloquialisms, and technical farming terminology, and can resolve issues by asking questions and interpreting answers appropriately. EAs also help with confidence. Farmers are naturally wary of taking AI's advice on changing practices that have been used for generations. An EA that knows the farmers and their history and that can put the AI's responses into local context is more reliable.

To handle the problem of hallucinations and other types of false output, Digital Green uses Retrieval-Augmented Generation (RAG). Although RAG is conceptually simple—find relevant documents and create a prompt that asks the model to generate its response to them—in practice, it is more complex. As anyone who has done a search knows, search results can give you a few thousand results. Incorporating all of these results into a RAG query would be impossible with most language models and impractical with the few that allow large context windows. So search results need to be scored for relevance. The most relevant documents need to be selected; Then the documents must be cut so that they contain only relevant parts. Keep in mind that for Digital Green, the issue is both multilingual and multimodal: relevant documents may change in any of the languages ​​or modes they use.

Every step of this pipeline must be carefully evaluated: translation software, text-to-speech software, relevance scoring, document harvesting, and the language model itself: Could another model do a better job? Guardrails need to be installed at every step to prevent false results. Results need to pass human review. Digital Green Tests with “Golden QAs” Top rated sets of questions and answers. Can the application consistently produce results as good as the “golden answer” when asked the “golden question”? Such testing needs to be done continuously. Digital Green also manually reviews 15% of their usage logs, to ensure their results are consistently of the highest quality. In his podcast for O'Reilly, Andrew Ng recently noted that the evaluation phase of product development often doesn't get the attention it deserves, in part because AI software is so easy to write. Is. Who wants to spend months testing an application that took a week to write? But that's what it takes to be successful.

Farmer.Chat is designed to be gender inclusive and climate smart. As 60% of the world's smallholder farmers are women, it is important for the application to welcome women and not assume that all farmers are men. Pronouns are important. So are role models; Farmers who present techniques in video clips and answer questions should include men and women.

Climate smart means making climate-sensitive recommendations wherever possible. Climate change is a huge issue for farmers, especially in countries like India where rising temperatures and changing rainfall patterns can be devastating. Recommendations should assess current climate patterns and the ways in which they may change. Climate smart recommendations are also less expensive. For example, while Farmer.Chat isn't afraid to recommend commercial fertilizers, it emphasizes local solutions: almost every farm can have an unlimited supply of fertilizer—which costs less than compost and uses agricultural waste. Helps to manage.

Farming can be very tradition-bound: “We do it because that's what my grandparents did and their parents before them.” A new farming technique coming from some faceless scientist in an urban office means little. If you hear that it has been used successfully by a farmer you know and respect, you are more likely to adopt it. To help farmers adopt new practices, Digital Green prioritizes peer-to-peer work whenever possible, using videos collected from local farmers. They try to keep farmers in touch with each other, celebrate their achievements and help farmers adopt new ideas.

Finally, there are both Farmer.Chat and FarmStack. open source. Software licenses may not directly affect farmers, but they are important in building a healthy ecosystem around projects that aim to do good. We see many applications that aim to monopolize the user's attention, subject the user to unwanted surveillance, or curtail political debate. An open source project to help people: We need more of this.

Over its history, of which Farmer.Chat is just the latest chapter, Digital Green has helped more than 6.3 million farmers, increased their income by 24%, and increased crop yields by 17%. Is. Farmer.Chat is the next step in this process. And we are surprised: the problems faced by small-scale farms in developed countries are not different from those in developing countries. Climate, insects and crop disease have no respect for economics or politics. Farmer.Chat helps small-scale farmers succeed in developing countries. We need the same services in the so-called “first world”.