Tech firms double down on multimodal LLMs

Chinese technology companies are doubling down on artificial intelligence-powered multimodal large language models, as part of a broader push to bolster the use of cutting-edge technology in a wider range of fields.

Multimodal LLMs boast the ability to process and generate various types of content, covering text, images, audio and video.

Experts said multimodal LLMs will lead the way in the further development of the generative AI industry, with significant potential for application in diverse industries such as finance, retail, healthcare and intelligent manufacturing.

Chinese internet heavyweight ByteDance recently released its latest AI model Doubao 1.5, which is equipped with deep thinking and vision comprehension capacities, while updating its text-to-image model to offer users imaging and visual content experience.

The newly launched model has received a significant upgrade in categories like mathematics, programming, scientific reasoning and creative writing, and has significantly reduced training and inference costs, the company said.

With visual reasoning ability, the model can ponder on what it sees. For instance, it can analyze landforms based on uploaded photos, help travelers conveniently choose restaurants while traveling and provide assistance to enterprises in project management and flowchart generation, so as to improve work efficiency and decision-making quality.

Lu Yanxia, research director at market research company International Data Corp China, said Chinese tech companies' technological advancements in multimodal LLMs will further promote the popularization of AI models, and bring fresh business opportunities for domestic AI servers, cloud computing and chip companies.

Such LLMs necessitate higher demand for data and knowledge in professional fields, and for talent that can fine-tune specialized models based on specific industrial demands, she said.

Chinese video-sharing platform Kuaishou Technology recently launched its newest Kling AI 2.0 video generation model. Since its launch in June last year, the Kling AI model has undergone over 20 iterations, with the number of global users surpassing 22 million.

The text-to-video AI model outperformed its rivals such as OpenAI's Sora in dimensions including semantic responsiveness, and visual and motion quality, Kuaishou claimed.

Gai Kun, senior vice-president and head of the community science department of Kuaishou, said AI holds immense potential for assisting creative expression, but some challenges persist in terms of the stability of AI-generated content, or AIGC, and the precise expression of users' complex creative ideas.

Gai said it is necessary to comprehensively enhance AI models' capabilities and improve human-machine interaction levels to "tell good stories with AI", adding that the rapid development of AIGC is reshaping many industries, such as advertising, film, television, entertainment and creativity.

Moreover, multimodal editing capabilities are currently available on Kling AI platform, where users can input their ideas through images and other formats, generating creative videos that align with their concepts, according to Kuaishou.

Over 15,000 developers worldwide have applied Kling API or application programming interface in various industrial scenarios, generating about 12 million images and over 40 million videos. Image-generated videos account for about 85 percent of Kling AI's video creation.

Wang Peng, an associate researcher at the Beijing Academy of Social Sciences, said the multimodal capabilities enable AI models to understand and process complex information more comprehensively, with wide application prospects in fields such as finance, intelligent customer service and healthcare.

Pan Helin, a member of the Expert Committee for Information and Communication Economy, which operates under the Ministry of Industry and Information Technology, said, "The training of multimodal AI models necessitates higher requirements for computing capacity, algorithms and high-quality data, and more efforts are required to bolster the efficient circulation of data elements, and expand application scenarios."

Pan emphasized that Chinese tech companies should improve independent innovation abilities in computing power chips and programming software, and invest more in basic scientific research, including mathematics, statistics and computer science to catch up with leading foreign counterparts amid intensifying international competition.