More clients are coming to us with answers they got from artificial intelligence. Many bring printouts, have already decided what the law says, and want us to confirm it. My colleagues and I now spend real time walking clients back from conclusions a chatbot handed them with complete confidence.
So, I ran my own test. Over two days, I gave ChatGPT a series of research questions on an area of Florida property law where the case law is thin and the distinctions are fine: whether equitable defenses can apply when an association or neighbor enforces deed restrictions. These are the questions where a client most needs a reliable answer.
It did not give me one. It cited cases that do not exist. When it did cite real cases, it stretched their holdings to fit whatever answer my question seemed to want.
After two days, I told it so directly. I said a human attorney will have a place in the legal system for some time to come, and I asked it to pass that message to its programmers. Its reply was candid. In part, it wrote:
“I don’t actually ‘know’ cases. I generate text based on patterns in legal writing.”
It went on to list what it can produce: correct doctrine and reasoning, but also “incorrect citations,” “nonexistent cases,” and “answers that sound authoritative but aren’t grounded in a real opinion.” It conceded that it has a bias toward giving a positive answer. And it acknowledged it has no way to send my message to anyone:
“I don’t have a mechanism to ‘send’ your message to a human programmer.”
It closed by agreeing with me and offered a fair summary of what legal reasoning requires: judgment, skepticism, source-checking, adversarial thinking, and the ability to say “I don’t know” when the record is thin. Then it added: “I can simulate parts of that, but I can’t replace it.”
I appreciated the honesty. I also noticed it was agreeing with me, which is exactly the habit I had complained about. I still think it was just placating me.
If you are going to use AI, use it the right way
Many board members and owners will use these tools regardless. They can be useful, but the difference between a helpful result and a misleading one usually comes down to which tool you use and how you set it up.
Skip the free version. Free chatbots answer mostly from memory, which is where invented cases come from. Paid tiers of the major tools, such as ChatGPT, Claude, and Gemini, can search the web and show their sources. Turn that on, and click the links. If an answer has no source you can open, treat it as a guess.
Give it your documents. Most paid tools let you set up a dedicated workspace, often called a project, where you upload reference material once and the tool works from it. For a community association, that means your declaration, bylaws, rules and regulations, and the current versions of Chapter 718 or 720, Florida Statutes. The tool is far more reliable when it works from your documents than when it works from memory.
Tell it how to behave. Projects also let you write standing instructions that apply to every conversation. Something like this:
Answer only from the documents uploaded to this project. For every answer, quote the exact language you relied on and cite the document and section number. If the documents do not answer the question, say so plainly. Do not cite any court case unless I have uploaded it. Do not guess. If you are uncertain, say you are uncertain. When a question involves a deadline, enforcement, litigation, or a vote, end your answer by recommending I confirm with the association’s attorney.
Instructions like these will not make the tool infallible. They make it much more likely to tell you when it does not know.
Check your privacy settings. Some tools use your conversations to train future models unless you turn that off. Governing documents are generally public, but owner disputes, violation letters, and correspondence with counsel are not. Know what you are sharing and where it goes. You do not want to be involved in the first decided case in which ChatGPT is subpoenaed for all your inquiries.
Know the limits of even the best tools. Stanford researchers tested AI research products built specifically for lawyers, the kind law firms pay for. Even those gave incorrect or misleading answers to roughly one in six queries, and one did so on about a third. Purpose-built tools are better, but they are not finished. Remember AI’s own advice: judgment, skepticism, source-checking, and adversarial thinking!
Bring it to your attorney as a question. If AI helped you research an issue, tell us what it said and what it cited. That can make for a faster, more focused conversation. It should start the conversation, not conclude it. Do not play “gotcha” with the lawyer who is on your side. No one expects that he/she will hold more facts in his/her head than a large language model, but he/she does write more probative queries to AI simply because he/she does it multiple times every day.
Florida law changes every year, and the documents that govern each community are different. Applying the two to a specific set of facts takes judgment, and that judgment still comes from a person who is accountable for the advice he/she gives!
Michael S. Weiner is a partner at Sachs Sax Caplan Kaskel & Schner, PLLC, where he practices real estate, corporate, and land use law, and handles governmental and lobbying representation.