A Practical Guide to Multimodal AI for Creative Work
Multimodal AI works with more than one kind of input, such as text and an image, and can return an answer or, in some tools, new media. For creative work, that might mean asking about a reference photo, turning a spoken note into a brief, or checking a sketch against written requirements. The useful question is what a particular tool accepts and returns, not whether its label says “multimodal”. Features vary by model and interface, so check the current documentation before building a workflow around them.
It helps to treat the model as one part of the process. You bring the source material, context and decisions; the model can help with a bounded task; you review the result before using it. That keeps the work practical and makes errors easier to catch.
In a small project, this can be as simple as pairing a photo with a question about its colour, layout or mood.
What multimodal AI means in practice
A text-only tool receives words and responds with words. A multimodal tool can work with a combination of text and media. Depending on the product, that may include images, audio, video or documents. For example, Google’s Gemini API documentation describes image and audio inputs, while OpenAI’s image-input guidance explains how images can be added to a ChatGPT conversation. These are examples of particular products, not a promise that every model supports the same files or tasks.
Input and output are separate capabilities. A tool that can inspect a photograph may only reply in text; another may generate an image but not accept audio. Before you start, check the supported file types, size limits, output formats and any plan restrictions. If a feature is important to your project, test it with a harmless sample first.
One tool may describe what it sees; another may turn a reference into a new draft, so the same upload can produce very different kinds of help.
Give it a clear creative brief
A useful prompt tells the model what the material is, what you want, and what should stay unchanged. Keep the first request narrow. For example, when sharing a sketch, ask for three possible colour directions and a short explanation of each. You can then choose one direction and request a revision, rather than asking for a finished design in one step.
“Look at this rough poster sketch. Suggest three colour palettes for a calm, coastal feel. Keep the layout and subject unchanged. Describe each palette in plain language, and point out anything you cannot confidently identify.”
For audio, say whether you need a transcript, a summary, or feedback on a specific passage. For a video, name the moments or details you want reviewed. If the output needs a particular shape—such as a shot list, table, caption set or checklist—ask for that format explicitly. Clear limits make it easier to compare the answer with your brief.
Review the result before you use it
Multimodal tools can miss small details, misread text in an image, or make a confident-sounding guess. OpenAI’s guidance for ChatGPT image inputs, for instance, calls out cases where image interpretation may be less accurate. Treat generated descriptions, captions and suggestions as drafts. Compare them with the original, check names and numbers, and ask a person with relevant knowledge to review anything that matters.
Check your source material as well. Use images, recordings, fonts and other assets you have permission to share and reuse. Read the provider’s current privacy and data controls before uploading client work, unpublished designs or personal information. If you cannot share an asset with a third party, describe the issue in general terms or use a local tool instead.
“Perplexity” has a specific technical meaning
Perplexity is a metric used to evaluate certain language models: it reflects how well a model predicts a sequence of tokens. It is not a general label for confusion in multimodal systems, and it is not a universal quality score for images, audio or creative usefulness. Comparisons also depend on tokenisation and evaluation setup. For a creative project, judge the output against your brief and the original material rather than treating one model metric as a verdict.
A simple workflow for your next project
- Choose one task, such as describing a reference or summarising a voice note.
- Check that the tool supports your file and that you can share it safely.
- Provide a short brief with the goal, constraints and requested format.
- Review the answer against the source; correct errors before reusing it.
- Save the prompt and your edits if you expect to repeat the task.
For hands-on creative work, try our Hex Code Finder to inspect colours in an image, the Meme Generator to add your own captions to an image, or the Music Generator to sketch a short MIDI idea. These tools are separate from AI models, but they offer a practical way to work directly with visual and audio material.