Your cousin is in a property dispute. He types his question into a chatbot and gets an answer in seconds. The answer names a section of the law and a court case. It sounds confident and it sounds complete. He is ready to act on it.
One problem. The case does not exist.
This is not a made up warning. It happened to lawyers in a New York court, and it can happen to anyone using legal AI in Pakistan. The words are smooth. The law can be wrong.
Where the idea of legal AI first went wrong in Pakistan
In March 2023, a judge in Phalia, a town in Mandi Bahauddin district, asked ChatGPT for help with a bail application. The case involved a 13 year old boy. The judge put the chatbot's answer into his written order.
He also wrote that his decision was his own, not the chatbot's. Gulf News reported the story at the time. Even then, the limits were visible. The chatbot itself said it could not see Pakistani case law and its knowledge stopped at a fixed date.
That single experiment opened a national debate. The debate ended with rules.
What the Supreme Court of Pakistan said about AI
In March 2025 the Supreme Court settled the direction in Ishfaq Ahmed v. Mushtaq Ahmed, reported as PLD 2025 SC 582. The 18 page judgment was written by Justice Syed Mansoor Ali Shah.
The Court said AI can help with legal research, drafting, translation, and case management. It said AI cannot decide a case, weigh evidence, or write the reasoning of a judgment. The Court warned about automation bias and about hallucinations, where the tool produces made up information. It said delegating the core work of judging to AI would amount to misconduct and would violate due process. The News covered the ruling in detail.
This warning was not abstract. AI that invents case law is a real problem we wrote about before, and the Court named it by name.
The existing law backs the Court's view. A judgment must be written and signed by the judge who presided over the case. A machine cannot sign a judgment. Human reasoning is the point of a court.
The national guidelines: a human stays in the loop
In 2026 the National Judicial (Policy Making) Committee issued the National Guidelines for the Use of AI in Judicial Institutions of Pakistan. The core rule is simple. A human stays in the loop. Every point where AI helped, the judge must record independent reasoning. The guidelines are published by the High Court of Sindh.
AI is allowed for scheduling, translation, transcription, and research support. It is not allowed to stand in for the judge.
The judiciary also built its own tool. Judge-GPT runs at the Federal Judicial Academy in Islamabad. It is trained on Pakistan's own statutes and case law and used by more than 1,500 district judges. Its standing rule: verify every reference it gives. Legalise.pk explains the full picture, including the National AI Policy 2025.
What one famous case teaches about AI mistakes
The most cited example comes from New York. In June 2023 a lawyer filed a court brief with six cases that ChatGPT had made up. The court checked the citations. The cases did not exist. The lawyer and his firm were fined 5,000 dollars. The sanctions order is public record.
If a trained lawyer can be burned by a confident chatbot, a person without a lawyer is in more danger. The chatbot does not know it is wrong.
Why privacy matters even more in Pakistan
Pakistan still has no data protection law. The bill drafted in 2023 remains a draft. DLA Piper's country page confirms the position.
That means no local rules tell an app how it may store or use the personal details you type in. Do not paste your CNIC number, your case documents, or your family matters into a free chatbot. The tool that helps you research is also collecting what you give it.
What to check before you trust a legal AI
Use this list before you act on anything an AI tells you.
- Ask it to show the actual law. Not a summary. The section text, in the original.
- Open its source yourself. If it names a case or a section, find it. If you cannot find it, do not use it.
- Ask how current it is. Laws change. An answer that was right in 2023 can be wrong today.
- Watch how it handles doubt. A legal AI that never says it is unsure is dangerous. The reliable ones tell you when to see a lawyer.
- Keep your private details out. Without a data protection law, your data has no local shield.
- For anything real, show a lawyer before you act. Money, freedom, and family are too heavy for a chatbot. And remember, what counts as evidence in a Pakistani court is decided by the court, not by an app.
If an AI drafts a document for you, the document is only the start. Whether an online agreement holds up in Pakistan is a separate legal question.
What this means for you
The same rule the Supreme Court gave the courts applies to you. Use legal AI to understand your problem. Use it to learn the right questions. Do not let it make the decision.
The decision belongs to a person who can be held responsible. A lawyer. A judge. And any tool you trust should show its work, with the actual law behind every answer.
Courts in Pakistan are also feeling the pressure. The Law and Justice Commission recorded 2,270,584 pending cases as of June 2025, which is why the judiciary is turning to AI at all. But the answer to delay is not a machine that invents law. It is a machine that helps a human work faster and better.
If you want to see where this is heading, read our guide on AI in Pakistan's courts and what is real today.
When an algorithm decides about you, there are rules that protect you. See your rights when AI makes the decision in Pakistan.
Using ChatGPT itself is not banned in Pakistan. See is ChatGPT legal in Pakistan for the rules that matter.
Browse more legal guides on the Lexiber blog, or start from the home page.Ask Lexiber free in English or Urdu. It shows you the exact section behind every answer.
