Project Description. This Project is a series of mini-studies (January – July 2023), thematically tied to AI evaluation of text-generators like ChatGPT, Microsoft Bing AI Chatbot, Google Bard Chatbot, and others. This research is in response to the recent release of the ChatGPT and other text-generating applications within Artificial Intelligence (AI) for the purpose of Natural Language Processing (NLP) using Machine Learning (ML) and other methods. NLP-based AI-enabled systems mimic natural language interactions with computer users, produce plausible coherent narratives in responses to extensive queries. ChatGPT’s user interface ease of use has served as a proof of concept for demonstrable qualitative jump in the systems capabilities. Its release has captured public attention, as evident in frequent mainstream media reports, and hype on social media. Rapid responses from major hi tech competitors, such as Microsoft and Google, testify to the companies to capitalize on the newly-generated attention and claim their stakes within pre-existing systems such as search engines and possibly soon in word processing packages). What is known in NLP and AI as Speech Generation, implemented via interactive interfaces, aka ChatBots or Conversational Agents, is a “hot topic” as it shows increased promise of its capabilities and its fast pace of adoption by the general public. These interfaces have been widely distributed to the general public, who no doubt are interested in harnessing its benefits, as well as finding out its drawbacks. This new development is fuelling the resurgence in human vs. AI capabilities debate. Our team of researchers will experiment with systems’ input and output to assess and predict their value, acceptance, impact, capabilities, and/or its predicted future, etc. as a line of research.
Project Team: Dr. Victoria Rubin, Alex Mayhew, Nicole Delellis, Dominique Kelly, Sarah Cornwell, Sodiq Onaolapo, and Dr. Yimin Chen (See Students Page for bios).
Suggested Citation: Rubin, Victoria L. (12 February 2023). Assessments of Text-Generating AI (2023). Project Description. The LiT.RL Project, University of Western Ontario. Available at https://victoriarubin.fims.uwo.ca/research/projects/assessments-of-text-generating-ai-2023/
Series of Mini-Studies
See Mini-Study 1. ChatGPT Assessment for Assignment Plagiarism. Open Working Notes (Unpublished)
Mini-Study 2: ChatGPT Media Coverage Metrics: Initial Examination (2023) by Delellis, Nicole, S., Chen, Yimin; Cornwell, Sarah, E., Kelly, Dominique; Mayhew, Alex; Onaolapo, Sodiq; and Rubin, Victoria, L.
[To be presented at ASIS&T 2023 in London, UK]
ABSTRACT: This paper presents an overview of coverage of OpenAI’s ChatGPT in media outlets from November 2022-March 2023, a comparison to previous media coverage of the chatbot Tay across the same outlets, and a count of ChatGPT media articles pertaining to government legislation and regulation. The New York Times, Wired, Gizmodo, The Globe and Mail, and The Guardian were searched for coverage. Across all five outlets there is an uptick in media coverage surrounding ChatGPT, with total numbers of included articles per month being 0 in November, 39 in December, 68 in January, 104 in February, and 143 in March. Findings exemplify the trend of increased coverage of ChatGPT in media public discourse, which contrasts with previous smaller media coverage of Tay. Examination of headlines and subheadings of included articles reveals minimal coverage (5.7%) dedicated to government legislation of ChatGPT. Future research will evaluate what is being said about ChatGPT within these media outlets.
KEYWORDS: ChatGPT; Media Coverage; Tay; Legislation
SUGGESTED CITATION: Delellis, Nicole, S., Chen, Yimin; Cornwell, Sarah, E., Kelly, Dominique; Mayhew, Alex; Onaolapo, Sodiq; and Rubin, Victoria, L. (2023). “ChatGPT Media Coverage Metrics: Initial Examination.” In the Proceedings of the the 86th Annual Meeting of the Association for Information Science & Technology | Oct. 29 – 31, 2023 | London, United Kingdom.
Mini-Study 3: Bing Chat: The Future of Search Engines? (2023). by Kelly, Dominique; Chen, Yimin; Cornwell, Sarah, E.; Delellis, Nicole; Mayhew, Alex; Onaolapo, Sodiq; and Rubin, Victoria, L.
[To be presented at ASIS&T 2023 in London, UK]
ABSTRACT: Introduced by Microsoft in February 2023, Bing Chat is a feature of the Bing search engine that integrates an OpenAI large language model (LLM) customised for search (Mehdi, 2023a). This poster compares the outputs of Bing Chat and a standard existing search engine (DuckDuckGo) in response to identical keyword queries and corresponding natural language (NL) questions. Specifically, we examined: (1) the length of Bing Chat’s responses and DuckDuckGo’s first page of search results, by number of website links; and, (2) the length of Bing Chat’s textual summaries, by number of website links. We found that, on average, significantly fewer websites were linked to in Bing Chat’s responses compared to DuckDuckGo’s search results. Our findings have important implications for website operators, who may receive less traffic and ad revenue if LLM-enabled search engines are widely adopted in the future. Human-Computer Interaction (HCI) will inevitably face the need for more research on human information behaviours adaptations in response to the changing search paradigm.
KEYWORDS: Search Engines; Bing Chat; DuckDuckGo; Large Language Models; ChatGPT
An example of a Bing Chat response from this study’s dataset.
SUGGESTED CITATION: Kelly, Dominique; Chen, Yimin; Cornwell, Sarah, E.; Delellis, Nicole; Mayhew, Alex; Onaolapo, Sodiq; and Rubin, Victoria, L. (2023). “Bing Chat: The Future of Search Engines?” In the Proceedings of the the 86th Annual Meeting of the Association for Information Science & Technology | Oct. 29 – 31, 2023 | London, United Kingdom.
Study 4: (Upcoming): it’s about “Students’ Behavioral Intention to Use Text-Generating Artificial Intelligence Technologies (ChatGPT, GPT-4, etc.) for Academic Activities”
[In Preparation in Fall 2023-Winter 2024]