We gave the generative AI language model a quick whirl to see how it could help us in our work.
February 16, 2023
Reading time 5min
ChatGPT is all the rage these days. Launched in November 2022, it is a language model developed by OpenAI that uses deep learning to generate text. It’s been touted to generate high-quality content in a fraction of time compared to traditional methods.
According to OpenAI CEO Sam Altman, it gained 1 million users in five days. Compare that to Facebook, which crossed the same mark in 10 months.
Naturally, we got curious and wanted to find out if it could be useful for our content strategy work.
PebbleRoad has a team of content strategists who help clients to create, deliver and govern useful and usable content. Some of our tasks include:
Primary and secondary research to understand business goals, user needs, current behaviours and more.
Information architecture and content modelling to improve findability, consistency, reusability, searchability and content management.
Style guides to define the voice and tone of the organisation and ensure that content owners can create quality content at scale.
Content design to answer user needs most effectively (e.g. through calculators or tabs instead of dense paragraphs).
Copyediting to simplify complex language and processes.
Putting ChatGPT to task
Would ChatGPT be able to replicate the work we do or at least help us do our work more quickly? We ran a quick experiment by asking it to do some of the tasks we normally do:
Propose voice and tone guidelines based on sample text and scenarios
Propose content style guidelines for tabs and toggles
Rewrite an existing page
We found that ChatGPT was great for:
Brainstorming, first drafts and rephrasing: The tool helped us to brainstorm interview questions efficiently and suggested questions we had not thought of! It could also propose possible fields, field types, instructions and character limits for our sample content model.
Drafting voice and tone guidelines: The tool was able to propose some useful voice and tone guidelines based on sample text AND a scenario. More specific guidelines should still be crafted in consultation with internal stakeholders and tested with users, though.
Simplifying technical terms in plain English: ChatGPT excels at explaining complex terms and ideas. This can cut editing time significantly and empower subject matter experts to simplify content at the source (instead of going through rounds of edits and discussions).
But from our experience, it seemed to struggle with:
Editing longer copy: Other AI tools such as Writer might be better at applying and managing multiple rules from your content style guide. Especially if you are applying those rules to longer pieces of information.
Content design: As it is still a text-based tool, ChatGPT might not be able to propose the best content patterns for consumption. For example, it struggled to propose more unique and complex patterns such as calculators, tabs and toggles
Content style guidelines: As of the time of writing (February 2023), ChatGPT seemed to conflate design, development and content guidelines. Though some of the guidelines might be relevant, teams should definitely refine their guidelines based on experience and best practices.
Interacting with ChatGPT
Our interactions have taught us that we need to ask the right questions, along with the appropriate inputs to get more out of ChatGPT.
An important thing to bear in mind is that while trillions of words have been fed into ChatGPT, its training data only goes up to 2021. (Accurate as of February 2023) This means it isn’t aware of more recents events, new discoveries or research breakthroughs. For instance, it doesn’t have the latest on the war in Ukraine. ChatGPT Plus, the paid version, however might offer new features and improvements.
Also, as its training data is drawn solely from the Internet, which can contain inaccuracies or misinformation, it’s super important to verify what it churns out by checking multiple sources!
As computer scientist Jonathan May puts it: “... it doesn’t have facts, per se. It just knows what word should come next… ChatGPT doesn’t try to write sentences that are true. But it does try to write sentences that are plausible.”
How to get the best out of ChatGPT
(As advised by ChatGPT itself)
Be clear and specific: Ask direct, well-defined questions to get the most accurate and helpful answers. P/S: Try this prompt that gets ChatGPT to write its own prompts!