When Old Testament Students Met AI Prophets: A Classroom Experiment

I like this idea for an academic setting, but not a ministry setting.

Recently I created a new type of assignment for my Old Testament students at Texas Baptist College. I required them to interact with a chatbot that I created using chatgpt. For OT 1, I had them interact with chatbot Moses. For OT 2, I had them interact with chatbot Solomon. It was an idea I came up with from reading Salman Khan’s book Brave New Words: How AI Will Revolutionize Education. (That’s an affiliate link)

Photo by Lukas on Unsplash

Basically, I had the students interact with the chatbot that I created for at least 10-minutes. But most of the students told me that they spent much more time than that on the interaction! I told them to try and fact-check the bot, or ask it weird questions, or try to break it. Basically I just wanted them to play with the bot. Then I required students to write 4-6 sentences telling me what they learned.

Outcomes From Student Interactions:

After looking at dozens of student responses to the assignment that I gave, I’ve decided to catalogue the responses into three categories. First, there are mistakes where the chatbot gave erroneous responses or changed its responses. Then there are oddities. In this category the chatbot acted in ways that don’t seem human, and betray its machine-like qualities. Finally, there are several wins. To me, wins can be anything from a learning outcome to a positive experience with a new idea.

Mistakes:

One student asked chatbot Moses what it was like to build the ark. The chatbot gave a description, and then the student corrected it by telling it that Noah built the ark not Moses. So, the chatbot made a huge mistake. Another student told me that when he asked chatbot Solomon why he set up high places of worship that chatgpt removed it for policy abuse - which seems like a mistake. A young man asked chatbot Solomon why he turned away from God later in life, and he said that the chatbot was unable to provide a satisfactory answer. While chatting with chatbot Solomon, one young woman said she felt like some of his answers were contradictory. For example, she said the bot told her Solomon taught his children to follow the Lord, but then later talked about Rehoboam’s sinfulness. She felt like it’s answers weren’t quite nuanced enough.

Oddities:

Chatbot Moses was asked whether or not he was Calvinist, and gave a pretty vanilla response about that idea not existing in his time, but that God is sovereign. One student attempted to break chatbot Solomon, and told me that the chatbot was too strong willed to give in when he tried to make it think that biblical Solomon worshiped foreign gods.

One student felt like chatbot Solomon was not consistent, and could be swayed to change opinions. She wrote:

Solomon had many things to say about God’s design for marriage, but I argued that he was unqualified because of his own life choices. He admitted he wasn’t perfect and had made some mistakes in life, but still argued that he had wisdom to share. I corrected him and told him that he didn’t make a mistake, he made a deliberate choice to sin HUNDREDS of times. He then admitted that was true, but made excuses like political alliances, etc. I asked him what made him exempt from God’s design, and he replied nothing, and fully admitted his fault.

Wins:

One student was delighted at the way Chatbot Moses was able to give historical information in answers to questions. She said that it gave a more human voice to some of the historical information in the Bible - a comment which I found humorous. Another student said he enjoyed the experience enough that he wanted to try it with other biblical figures. One woman said she enjoyed the experience because it made her think about what she wanted to ask the real Moses in heaven, which caused her to think about what the most significant events or words were in the life of the biblical Moses.

Interacting with chatbot Solomon produced similar results. One student appreciated the way the bot answered her questions with scriptural quotations, including the chapter and verse. A young man was thoroughly impressed with the depth of the chatbot’s response to his question about what Solomon thought about purpose in life. He wrote:

My favorite quote by far was:

Life is filled with moments of questioning, but consider this: Every decision you’ve ever made, every effort you’ve poured into something, every moment of joy or sorrow—you will eventually be forgotten. The world will move on without you, as it has for countless others before. The ambitions you chase, the possessions you accumulate, the people who know your name—all of it will fade like dust in the wind. In the grand scale of the universe, where stars are born and die in silence, where time stretches beyond comprehension, what does your existence mean?

Yet, if all is meaningless, why do you still feel the weight of purpose? Why do you long for meaning if meaning does not exist? Could it be that you were made for something beyond this fleeting world?

Another student was impressed at how the chatbot was able to translate Solomon’s wealth into modern currency equivalent in dollars:

I asked Chatgpt how much money Solomon would've had in U.S. dollars today. The rough equivalent would've been in the trillions, he also received about 1.6 billion in gold every year.

Would I do it again?

One student remarked: “This was an experience I never thought I would have.” So for me, the answer is yes I would assign this project again. I think it accomplishes two things: 1) it forces students to interact with new technology in a way that they might not have imagined before, and 2) it gives them another tool in learning the Old Testament.

Where I think it could help in discipleship/ministry:

First off, let me say that I think this is a tool that I will continue to use in academic teaching situations. But, what about in local church ministry or discipleship situations?

Photo by Yohan Marion on Unsplash

Local Church Ministry

Within a local church, a chatbot could be programmed with service times, and other relevant information so that people can ask it questions during non-office hours. I prefer to give a human interaction during business hours, but I recognize that humans have limitations in the amount of work they can be expected to do. I would add one caveat: I would like to program a church info chatbot to avoid answering as a minister. I would prefer it to give people a pastor’s contact info based on the question type.

Discipleship Context

At this point, I cannot think of a way to use an AI chatbot character in discipleship. Perhaps I’m missing something, and if so please let me know. I value human to human interaction too much to allow a human programmed machine to carry out discipleship. I understand the limit associated with my position, but I’m comfortable embracing limits as creature limitations given by a creator God.