“TIBBIYOT OLIYGOHLARIDA TABIIY FANLARNI
INTERFAOL USULLARDA O'QITISHNING
MUAMMOLARI VA YECHIMLARI”
656
CHALLENGES IN REPRESENTING MULTIMODAL DISCOURSE IN CORPUS
LINGUISTICS: A CASE STUDY OF SOCIAL MEDIA COMMUNICATION
Dilnoza Tashpulatova Kholmirza kizi
Assosiate professor, Phd
University of Science and technologies
Department of Languages
E-mail:
dilnozatashpulatova05@gmail.com
Abstract
Social media platforms have transformed human communication by integrating multiple
semiotic modes—text, images, emojis, video, and audio—into a single discourse space. This
multimodal nature poses substantial challenges for corpus linguistics, a field traditionally focused on
textual data. This paper investigates the key methodological, technical, and theoretical challenges of
representing multimodal discourse in social media corpora. Through a case study approach, it
critically examines existing annotation frameworks, data management strategies, and multimodal
integration techniques. Finally, the paper proposes directions for developing richer, more
comprehensive multimodal corpora that reflect the complexity of contemporary digital
communication.
Keywords:
multimodal discourse, corpus linguistics, social media, annotation, data
integration, discourse analysis, digital communication.
Introduction
In recent years, social media communication has increasingly embraced multimodality—the
integration of multiple modes of meaning-making such as language, images, sounds, and gestures.
Unlike traditional monomodal corpora, social media posts often combine text with emojis, photos,
videos, GIFs, and hyperlinks, presenting unique challenges for corpus linguistics. The discipline,
historically oriented toward textual data, must now expand its methodological tools and theoretical
frameworks to adequately represent this diversity (Kress & van Leeuwen, 2001; O’Halloran, 2011).
This study explores these challenges through the lens of social media communication,
focusing on how multimodal discourse can be collected, annotated, integrated, and analyzed within
corpus linguistics. The paper aims to:
•
Identify core challenges in representing multimodal data;
•
Analyze existing annotation and integration frameworks;
•
Propose practical and theoretical strategies to enhance corpus representation
of multimodal discourse.
Background and Theoretical Framework
Multimodality refers to communication practices that deploy multiple semiotic modes
simultaneously (Jewitt, 2009). In social media, this includes text, images, videos, audio clips, emojis,
and hyperlinks, all contributing to meaning-making processes. Corpus linguistics traditionally relies
on large-scale text corpora to uncover linguistic patterns. However, multimodality demands more
complex representation models, capable of capturing interrelations among modes (Baldry & Thibault,
2006).
Social media platforms such as Twitter, Instagram, Facebook, and TikTok facilitate
multimodal communication by enabling users to embed images, videos, and other visual or auditory
elements alongside text (Zappavigna, 2018). The brevity and fragmentation of social media discourse
complicate the definition of units of analysis, further challenging corpus construction and annotation
(Page, 2019).
3. Methodological Challenges
Unlike traditional text corpora, social media corpora must accommodate diverse data types,
including multimedia files, emoji characters, and metadata such as hashtags and user mentions. This
“TIBBIYOT OLIYGOHLARIDA TABIIY FANLARNI
INTERFAOL USULLARDA O'QITISHNING
MUAMMOLARI VA YECHIMLARI”
657
requires robust scraping tools and storage infrastructures capable of handling heterogeneous data.
Moreover, privacy and ethical considerations limit data access and sharing (Wang et al., 2020).
A core challenge lies in the transcription of non-textual elements into a corpus-friendly format.
Emojis, for example, can be transcribed as Unicode characters, but images and videos require more
complex coding schemes. Existing multimodal annotation tools such as ELAN and ATLAS.ti offer
tiered annotation for video and audio but struggle to scale for large social media datasets (Selting et
al., 2009).
Integration of Multimodal Elements
Integrating multiple modes into a unified analytic framework remains a technical and
theoretical hurdle. Cross-modal alignment—linking textual elements with corresponding images or
gestures—requires sophisticated representational strategies. Approaches include multimodal fusion
models using machine learning (e.g., tensor fusion networks) but are still under development in
corpus linguistics (Zadeh et al., 2017).
4. Theoretical Challenges
Social media discourse is characterized by fragmented posts, replies, and hashtags. Deciding
the unit of analysis—whether individual posts, conversational threads, or multimodal ensembles—is
complex and impacts annotation and analysis (Page, 2019).
Multimodal discourse often relies on context, including shared cultural knowledge and
platform-specific norms (e.g., meme culture). Capturing these aspects in corpus annotation is
challenging but essential for valid interpretation (Seargeant & Tagg, 2014).
Case Study: Twitter Multimodal Corpus
Using the Twitter API, a dataset of 10,000 tweets containing text, emojis, and images was
collected over a one-month period. Metadata including hashtags, user mentions, and timestamps were
also retrieved.
•
Textual annotation:
Part-of-speech tagging, sentiment annotation
•
Emoji transcription:
Unicode representation plus sentiment classification
•
Image annotation:
Manual coding for content type (e.g., meme, selfie), affective
features, and interaction with text (using insertion/extension framework by Bateman et al., 2017)
•
Aligning emoji sentiment with textual sentiment proved inconsistent, highlighting
polysemy in emoji use.
•
Images often extended textual meaning but posed difficulties in standardizing
annotation categories.
•
Reply threads revealed the fragmented nature of discourse, complicating coherent
corpus structuring.
Discussion
This study reveals that the key challenges in representing multimodal social media discourse
stem from the heterogeneity of data, annotation complexity, and lack of standardized frameworks.
Multimodal corpora must balance methodological rigor with scalability. Incorporating machine
learning techniques alongside human interpretation offers promising avenues for future research.
Conclusion and Future Directions
The integration of multimodal discourse into corpus linguistics represents a frontier with
significant challenges and opportunities. Advances in annotation standards, data integration tools,
and theoretical models are essential to capture the complexity of social media communication. Future
work should prioritize interdisciplinary collaboration, ethical data handling, and the development of
scalable multimodal corpora.
References
1.
Baldry, A., & Thibault, P. J. (2006).
Multimodal transcription and text analysis: A
multimedia toolkit and coursebook
. Equinox Publishing.
2.
Bateman, J., Delin, J., Henschel, R., & Wildfeuer, J. (2017). Representing multimodal
discourse: The insertion/extension model.
Discourse Studies
, 19(6), 659-688.
“TIBBIYOT OLIYGOHLARIDA TABIIY FANLARNI
INTERFAOL USULLARDA O'QITISHNING
MUAMMOLARI VA YECHIMLARI”
658
3.
Jewitt, C. (2009). An introduction to multimodality. In C. Jewitt (Ed.),
The Routledge
handbook of multimodal analysis
(pp. 14–27). Routledge.
4.
Kress, G., & van Leeuwen, T. (2001).
Multimodal discourse: The modes and media
of contemporary communication
. Arnold.
5.
O’Halloran, K. (2011). Multimodal discourse analysis and digital technology.
Continuum
.
6.
Page, R. E. (2019).
The linguistics of social media
. Edinburgh University Press.
7.
Seargeant, P., & Tagg, C. (2014).
The language of social media: Identity and
community on the internet
. Palgrave Macmillan.
8.
Selting, M., et al. (2009). Conversation analysis and multimodality: Video and audio
annotation and transcription.
Discourse Studies
, 11(5), 571-590.
9.
Wang, Y., et al. (2020). Ethical considerations in social media research.
Journal of
Empirical Research on Human Research Ethics
, 15(3), 236-242.
10.
Zadeh, A., et al. (2017). Tensor fusion network for multimodal sentiment analysis.
EMNLP 2017
.
11.
Zappavigna, M. (2018).
Searchable talk: Hashtags and social media metadiscourse
.
Bloomsbury.
12.
Tashpulatova, D., & Siddiqova, I. (2021, April). A CRITIQUE OF THE VIEW OF
ANTONYMY AS A RELATION BETWEEN WORD FORMS. In
Конференции
.
13.
Тошпулатова, Д. Х. (2019). ПОНЯТИЕ И ВИДЫ ПЕДАГОГИЧЕСКИХ
КОНЦЕПЦИЙ ОБУЧЕНИЯ ИНОСТРАННОМУ ЯЗЫКУ. АНАЛИЗ СОВРЕМЕННЫХ
ПЕДАГОГИЧЕСКИХ КОНЦЕПЦИЙ.
Вопросы педагогики
, (7-2), 118-121.
14.
ТАШПУЛАТОВА, Д. PRAGMATIC APHORISMS IN UZBEK AND ENGLISH
FEATURE
AND
THE
PRINCIPLES
OF
THEIR
TRANSMISSION
IN
THE
CORPUS.
СООТНОШЕНИЕ ПАРАЛИНГВИСТИКИ И РЕЧЕВОГО ЭТИКЕТА В РАЗНЫХ
ЛИНГВОКУЛЬТУРАХ
.
15.
qizi Turaboyeva, S. Z., & qizi Tashpulatova, D. X. (2022). O’ZBEK VA INGLIZ
TILLARIDAGI AXLOQIY QADRIYATLAR MAZMUNINI IFODALOVCHI BIRLIKLARNING
LINGVOKULTUROLOGIK XUSUSIYATLARI.
Academic research in modern science
,
1
(1), 143-
147.
16.
Ташпулатова, Д. Х. К., & Умурзакова, Б. Э. (2023). НЕРВНАЯ
СИСТЕМА.
Academic research in educational sciences
,
4
(TMA Conference), 303-309.
17.
Tashpulatova, D. K. K., & Jiyenbayeva, B. (2023). THE IMPACT OF LEARNING A
LANGUAGE ON THE BRAIN FUNCTION.
Academic research in educational sciences
,
4
(TMA
Conference), 310-314.
18.
NAVOI, A., & BABUR, Z. M. (2022). RESEARCH AND EDUCATION.
